commitf0b9b2af67Merge:373d65abbaa7bbAuthor: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Sat Sep 12 18:17:04 2026 +0800 Merge branch 'merge-testing' commit373d65a9d5Merge:28fc652dd0e6fdAuthor: harine <harinesumen@gmail.com> Date: Sat Sep 12 17:34:27 2026 +0800 Merge branch 'master' into Harine commit28fc652097Author: harine <harinesumen@gmail.com> Date: Sat Sep 12 17:34:07 2026 +0800 new stuff added yes commitbbaa7bb04fMerge:1c6ced2610bfd7Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Fri Sep 11 11:18:35 2026 +0800 Merge branch 'Gabriel' into merge-testing commit1c6ced2dceAuthor: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Fri Sep 11 11:12:30 2026 +0800 this good commitdd0e6fdc39Author: DefiantWanderer <yongwei1349@gmail.com> Date: Fri Sep 11 10:32:34 2026 +0800 Revert "fixing compatibility issues" This reverts commit564e7d83f8. commit564e7d83f8Author: DefiantWanderer <yongwei1349@gmail.com> Date: Fri Sep 11 10:28:40 2026 +0800 fixing compatibility issues commit610bfd7991Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Fri Sep 11 10:13:25 2026 +0800 Merged Khalil commitcfe76da1d3Merge:584818b9fcdc13Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Fri Sep 11 10:03:51 2026 +0800 Merge branch 'Khalil' into Gabriel commit584818b95aAuthor: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Thu Sep 10 12:47:06 2026 +0800 added back phase toggle button commit17fad430c1Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Thu Sep 10 12:09:46 2026 +0800 Cleaned up merge UI, added reset step button commitac4366d04dAuthor: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Thu Sep 10 00:31:47 2026 +0800 Added some UI to recording commitd601efcf43Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Thu Sep 10 00:13:06 2026 +0800 Corrected 5 step approach phases & added placeholders for future logic commit7cf217488aAuthor: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Wed Sep 9 22:48:06 2026 +0800 Cleared warnings commit981071ef74Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Wed Sep 9 20:09:18 2026 +0800 Fix Merge, cleaned up UI commitf30bbe2a34Merge:2f451157518594Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com> Date: Tue Sep 8 14:11:21 2026 +0800 Merge branch 'au-au' into Gabriel commit9fcdc13699Author: Khalil Belabadia <belabakhalil@gmail.com> Date: Mon Sep 7 22:16:00 2026 +0800 Audio feedback update Added the following files : - AudioCue.kt - AudioFeedbackEngine - AudioFeedbackSettings Edited files : - BowlingCameraActivity (added the triggerAudiofeedback function) commit7518594440Merge:e7a9c2b7af128dAuthor: midnight-masala <2401021@sit.singaporetech.edu.sg> Date: Mon Sep 7 21:26:18 2026 +0800 Merge remote-tracking branch 'origin/au-au' into au-au # Conflicts: # app/src/main/java/com/example/jnicpp/bowling/BowlingCameraActivity.kt # app/src/main/res/layout-land/activity_bowling_camera.xml # app/src/main/res/layout/activity_bowling_camera.xml commite7a9c2b140Author: midnight-masala <2401021@sit.singaporetech.edu.sg> Date: Mon Sep 7 21:22:11 2026 +0800 Add live per-step form feedback based on joint angles Extends pose angle tracking with knee bend (hip-knee-ankle), then uses it alongside the existing elbow/shoulder angles in a new PoseStageAdvisor to give a short live cue for whichever step of the approach is in progress: push-away on step 2, downswing on step 3, backswing on step 4, and knee-bend/arm-extension on the final step. Thresholds are starting defaults, not measured coaching data, and are expected to be retuned against real approach footage. Also fixes a layout bug found while testing this live: the new feedback text was chained via ConstraintLayout's toBottomOf to the step banner above it, so whenever that banner was hidden (GONE) the feedback text rendered at its collapsed zero-height position instead of staying put, landing on top of the recording indicator. Both now sit in a plain vertical LinearLayout, which collapses GONE children correctly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> commit7af128d575Author: midnight-masala <2401021@sit.singaporetech.edu.sg> Date: Mon Sep 7 20:58:03 2026 +0800 steps identify the steps commit2bb7c6ca7bAuthor: midnight-masala <2401021@sit.singaporetech.edu.sg> Date: Mon Sep 7 20:58:03 2026 +0800 Add live final-position feedback and fix step detection accuracy Shows a live banner as each step of the approach is counted, ending in "FINAL POSITION - RELEASE!" once the 5th footfall lands. Also fixes LiveStepDetector's peak-prominence check, which compared a candidate footfall only to its immediate neighboring frame and silently dropped real steps that landed across several closely-spaced frames; it now tracks the true rise/fall trough on each side instead. Switches PoseAnalyzer to ML Kit's faster base pose model (the accurate model was starving the peak detector of frames on unaccelerated hardware), and makes LiveStepDetector's stillness-based reset toggleable, currently off for easier testing against recorded reference clips. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
250 lines
11 KiB
Kotlin
250 lines
11 KiB
Kotlin
/**
|
|
* @file CameraViewModel.kt
|
|
* @brief UI/recording state machine and pose-data buffering for the bowling camera screen.
|
|
*/
|
|
package com.example.jnicpp.bowling
|
|
|
|
import android.app.Application
|
|
import androidx.lifecycle.AndroidViewModel
|
|
import androidx.lifecycle.viewModelScope
|
|
import kotlinx.coroutines.Job
|
|
import kotlinx.coroutines.delay
|
|
import kotlinx.coroutines.flow.MutableSharedFlow
|
|
import kotlinx.coroutines.flow.MutableStateFlow
|
|
import kotlinx.coroutines.flow.SharedFlow
|
|
import kotlinx.coroutines.flow.StateFlow
|
|
import kotlinx.coroutines.flow.asSharedFlow
|
|
import kotlinx.coroutines.flow.asStateFlow
|
|
import kotlinx.coroutines.isActive
|
|
import kotlinx.coroutines.launch
|
|
import kotlin.time.Duration.Companion.seconds
|
|
|
|
/**
|
|
* @brief Holds camera/recording UI state so it survives configuration
|
|
* changes and so the state machine lives outside the Activity.
|
|
*
|
|
* This class knows nothing about CameraX or ML Kit APIs directly --
|
|
* [BowlingCameraActivity] and [CameraXController] report events into it,
|
|
* and the UI observes it back out. That keeps this class trivially
|
|
* unit-testable (no Android camera framework involved).
|
|
*
|
|
* Extends [AndroidViewModel] rather than a plain ViewModel solely to reach
|
|
* an Application [android.content.Context] for [DetectorSettings.load] --
|
|
* see [onRecordingStarting].
|
|
*/
|
|
class CameraViewModel(application: Application) : AndroidViewModel(application) {
|
|
|
|
/** @brief The camera screen's overall recording state. */
|
|
sealed interface RecordingState {
|
|
/** @brief No recording in progress or starting. */
|
|
data object Idle : RecordingState
|
|
/** @brief A recording has been requested but hasn't started writing yet. */
|
|
data object Starting : RecordingState
|
|
/** @brief A recording is actively being written. @param elapsedSeconds Seconds elapsed since recording started. */
|
|
data class Recording(val elapsedSeconds: Long) : RecordingState
|
|
}
|
|
|
|
private val _recordingState = MutableStateFlow<RecordingState>(RecordingState.Idle)
|
|
/** @brief Current recording state, observed by the UI to drive button/timer/indicator visibility. */
|
|
val recordingState: StateFlow<RecordingState> = _recordingState.asStateFlow()
|
|
|
|
// Whether live pose detection/overlay is on. Only meant to change while
|
|
// recordingState is Idle -- the UI disables the toggle otherwise (see
|
|
// BowlingCameraActivity#renderRecordingState) since the recording
|
|
// pipeline picks its pose mode once at start.
|
|
private val _poseEnabled = MutableStateFlow(value = false)
|
|
/** @brief Whether pose detection/overlay is currently enabled. */
|
|
val poseEnabled: StateFlow<Boolean> = _poseEnabled.asStateFlow()
|
|
|
|
// Latest per-frame joint angles, for the overlay's angle-label text now
|
|
// and for frame-by-frame swing analysis/logging later. Null whenever
|
|
// there's no current pose result to derive them from (pose off, no
|
|
// frame processed yet, or a frame with no landmarks that met the
|
|
// confidence bar -- see PoseAngleCalculator).
|
|
private val _poseAngles = MutableStateFlow<PoseAngles?>(null)
|
|
/** @brief Joint angles computed for the most recent analyzed frame, or null if none available. */
|
|
@Suppress("unused")
|
|
val poseAngles: StateFlow<PoseAngles?> = _poseAngles.asStateFlow()
|
|
|
|
// Pose-frame buffering and live step counting for the current/most
|
|
// recent recording session -- see StepCountingSession's class doc for
|
|
// why this lives in its own class rather than inline here. Frames only
|
|
// flow into it while actually recording (see onPoseFrameUpdated) -- a
|
|
// live preview with pose overlay on but not recording doesn't feed it.
|
|
private val stepCountingSession = StepCountingSession()
|
|
/** @brief Time-ordered pose samples buffered for the current/most recent recording session. */
|
|
val poseFrames: List<PoseFrame> get() = stepCountingSession.poseFrames
|
|
|
|
// Live delivery-phase classification (starting stance, approach, etc)
|
|
private val posePhaseDetector = PosePhaseDetector()
|
|
private val _posePhase = MutableStateFlow<BowlingPhase?>(null)
|
|
//The bowler's current delivery phase, or null if no phase currently validates.
|
|
val posePhase: StateFlow<BowlingPhase?> = _posePhase.asStateFlow()
|
|
|
|
// Raw torso/knee/elbow angle readings behind posePhase above, for
|
|
// showing the bowler the actual numbers rather than just a pass/fail
|
|
// signal -- see PosePhaseDetector.Metrics.
|
|
private val _poseMetrics = MutableStateFlow<PosePhaseDetector.Metrics?>(null)
|
|
//This frame's torso/knee/elbow angle readings, or null if pose detection is off.
|
|
val poseMetrics: StateFlow<PosePhaseDetector.Metrics?> = _poseMetrics.asStateFlow()
|
|
|
|
/** @brief Steps detected so far in the current attempt, since the last reset. */
|
|
val stepEvents: StateFlow<List<StepEvent>> get() = stepCountingSession.stepEvents
|
|
|
|
// Live "how's my form right now" cue for whichever step is currently in
|
|
// progress -- see PoseStageAdvisor. Recomputed every frame alongside
|
|
// stepEvents so it's always tied to the same step count the UI already
|
|
// shows, and cleared on the same resets stepEvents is.
|
|
private val _poseStageFeedback = MutableStateFlow<String?>(null)
|
|
/** @brief Live form feedback for the current step, or null if there's nothing to say yet. */
|
|
val poseStageFeedback: StateFlow<String?> = _poseStageFeedback.asStateFlow()
|
|
|
|
private val _permissionsGranted = MutableStateFlow(value = false)
|
|
/** @brief Whether all required camera/microphone/storage permissions are currently granted. */
|
|
@Suppress("unused")
|
|
val permissionsGranted: StateFlow<Boolean> = _permissionsGranted.asStateFlow()
|
|
|
|
// One-shot user-facing error messages (camera unavailable, detector
|
|
// failure, storage failure, ...). SharedFlow, not StateFlow, so the same
|
|
// error doesn't get replayed and re-shown after a config change.
|
|
private val _errorEvents = MutableSharedFlow<String>(extraBufferCapacity = 4)
|
|
/** @brief One-shot user-facing error messages, e.g. for a Toast/Snackbar. */
|
|
val errorEvents: SharedFlow<String> = _errorEvents.asSharedFlow()
|
|
|
|
private var timerJob: Job? = null
|
|
|
|
/**
|
|
* @brief Records the result of a runtime permission request.
|
|
* @param granted true if all required permissions were granted.
|
|
*/
|
|
fun onPermissionsResult(granted: Boolean) {
|
|
_permissionsGranted.value = granted
|
|
}
|
|
|
|
/**
|
|
* @brief Toggles live pose detection/overlay on or off.
|
|
* @param enabled true to enable pose detection/overlay, false to disable.
|
|
*/
|
|
fun onPoseToggled(enabled: Boolean) {
|
|
_poseEnabled.value = enabled
|
|
if (!enabled) {
|
|
_poseAngles.value = null
|
|
posePhaseDetector.reset()
|
|
_posePhase.value = null
|
|
_poseMetrics.value = null
|
|
}
|
|
}
|
|
|
|
/**
|
|
* @brief Reports one analyzed frame's landmarks and joint angles.
|
|
*
|
|
* Always updates the live angle overlay. Buffering into [poseFrames]
|
|
* and live step counting are scoped to an actual recording (see
|
|
* [poseFrames]'s doc), so a preview with pose overlay on but not
|
|
* recording doesn't silently accumulate frames outside any session.
|
|
*
|
|
* @param landmarks EMA-smoothed landmarks for this frame, keyed by ML Kit's `PoseLandmark` type constant.
|
|
* @param angles Joint angles computed for this same frame.
|
|
*/
|
|
fun onPoseFrameUpdated(landmarks: Map<Int, SmoothedLandmark>, angles: PoseAngles) {
|
|
_poseAngles.value = angles
|
|
|
|
// Update pose phase detector with this frame's landmarks and angles (independent of step counter)
|
|
val phaseResult = posePhaseDetector.update(landmarks, angles)
|
|
_posePhase.value = phaseResult.phase
|
|
_poseMetrics.value = phaseResult.metrics
|
|
|
|
// Check if the current pose matches the Starting Stance (pure posture query)
|
|
val isStartingStance = (phaseResult.phase == BowlingPhase.STARTING_STANCE)
|
|
|| posePhaseDetector.isStartingStanceValid(phaseResult.metrics)
|
|
|
|
if (_recordingState.value is RecordingState.Recording) {
|
|
stepCountingSession.onFrame(
|
|
landmarks = landmarks,
|
|
angles = angles,
|
|
timestampMs = System.currentTimeMillis(),
|
|
isStartingPosition = isStartingStance,
|
|
)
|
|
val currentStepCount = stepEvents.value.size
|
|
_poseStageFeedback.value = PoseStageAdvisor.feedback(
|
|
stepNumber = currentStepCount.takeIf { it > 0 },
|
|
angles = angles,
|
|
)
|
|
}
|
|
}
|
|
|
|
/** @brief Manually resets step counting state for the current recording session. */
|
|
fun resetStepCounter() {
|
|
stepCountingSession.resetStepCounter()
|
|
_poseStageFeedback.value = null
|
|
}
|
|
|
|
/**
|
|
* @brief Marks a recording as being requested and resets all
|
|
* per-session buffering/detection state.
|
|
*
|
|
* Reloads [DetectorSettings] fresh here (rather than once at
|
|
* construction) so a tuning change made in [ParameterEditorActivity]
|
|
* takes effect on the very next recording, without needing to
|
|
* restart this screen.
|
|
*/
|
|
fun onRecordingStarting() {
|
|
_recordingState.value = RecordingState.Starting
|
|
stepCountingSession.startNewSession(DetectorSettings.load(getApplication()))
|
|
/*poseFrameBuffer.clear()
|
|
ankleHipSmoother.reset()
|
|
liveStepDetector.reset()
|
|
_stepEvents.value = emptyList()
|
|
_poseStageFeedback.value = null*/
|
|
}
|
|
|
|
/** @brief Marks a recording as actively writing and starts the elapsed-time timer. */
|
|
fun onRecordingStarted() {
|
|
timerJob?.cancel()
|
|
timerJob = viewModelScope.launch {
|
|
var seconds = 0L
|
|
while (isActive) {
|
|
_recordingState.value = RecordingState.Recording(seconds)
|
|
delay(1.seconds)
|
|
seconds++
|
|
}
|
|
}
|
|
}
|
|
|
|
/**
|
|
* @brief Marks the current recording as finished and stops the elapsed-time timer.
|
|
*
|
|
* [stepEvents] is left as whatever [LiveStepDetector] had already
|
|
* counted live -- it already reflects the last in-progress attempt's
|
|
* steps, and a fresh [StepDetector.detect] batch pass over the whole
|
|
* buffer here would ignore any mid-recording resets and overcount
|
|
* across separate attempts.
|
|
*/
|
|
fun onRecordingStopped() {
|
|
timerJob?.cancel()
|
|
timerJob = null
|
|
_recordingState.value = RecordingState.Idle
|
|
}
|
|
|
|
/**
|
|
* @brief Surfaces a one-shot error message to the UI, and clears a
|
|
* stuck recording indicator if one was in progress.
|
|
*
|
|
* A failed start/stop shouldn't leave the UI stuck showing a recording
|
|
* indicator that no longer reflects reality.
|
|
*
|
|
* @param message Human-readable error message to surface.
|
|
*/
|
|
fun postError(message: String) {
|
|
_errorEvents.tryEmit(message)
|
|
if (_recordingState.value != RecordingState.Idle) {
|
|
onRecordingStopped()
|
|
}
|
|
}
|
|
|
|
/** @brief Cancels the elapsed-time timer when this ViewModel is destroyed. */
|
|
override fun onCleared() {
|
|
timerJob?.cancel()
|
|
}
|
|
}
|