Files
PinPoint/app/src/main/java/com/example/jnicpp/bowling/CameraViewModel.kt
T
jingwen121 c6501d00f2 Squashed commit of the following:
commit f0b9b2af67
Merge: 373d65a bbaa7bb
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Sat Sep 12 18:17:04 2026 +0800

    Merge branch 'merge-testing'

commit 373d65a9d5
Merge: 28fc652 dd0e6fd
Author: harine <harinesumen@gmail.com>
Date:   Sat Sep 12 17:34:27 2026 +0800

    Merge branch 'master' into Harine

commit 28fc652097
Author: harine <harinesumen@gmail.com>
Date:   Sat Sep 12 17:34:07 2026 +0800

    new stuff added yes

commit bbaa7bb04f
Merge: 1c6ced2 610bfd7
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 11:18:35 2026 +0800

    Merge branch 'Gabriel' into merge-testing

commit 1c6ced2dce
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 11:12:30 2026 +0800

    this good

commit dd0e6fdc39
Author: DefiantWanderer <yongwei1349@gmail.com>
Date:   Fri Sep 11 10:32:34 2026 +0800

    Revert "fixing compatibility issues"

    This reverts commit 564e7d83f8.

commit 564e7d83f8
Author: DefiantWanderer <yongwei1349@gmail.com>
Date:   Fri Sep 11 10:28:40 2026 +0800

    fixing compatibility issues

commit 610bfd7991
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 10:13:25 2026 +0800

    Merged Khalil

commit cfe76da1d3
Merge: 584818b 9fcdc13
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 10:03:51 2026 +0800

    Merge branch 'Khalil' into Gabriel

commit 584818b95a
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 12:47:06 2026 +0800

    added back phase toggle button

commit 17fad430c1
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 12:09:46 2026 +0800

    Cleaned up merge UI, added reset step button

commit ac4366d04d
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 00:31:47 2026 +0800

    Added some UI to recording

commit d601efcf43
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 00:13:06 2026 +0800

    Corrected 5 step approach phases & added placeholders for future logic

commit 7cf217488a
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Wed Sep 9 22:48:06 2026 +0800

    Cleared warnings

commit 981071ef74
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Wed Sep 9 20:09:18 2026 +0800

    Fix Merge, cleaned up UI

commit f30bbe2a34
Merge: 2f45115 7518594
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Tue Sep 8 14:11:21 2026 +0800

    Merge branch 'au-au' into Gabriel

commit 9fcdc13699
Author: Khalil Belabadia <belabakhalil@gmail.com>
Date:   Mon Sep 7 22:16:00 2026 +0800

    Audio feedback update

    Added the following files :

    - AudioCue.kt
    - AudioFeedbackEngine
    - AudioFeedbackSettings

    Edited files :

    - BowlingCameraActivity (added the triggerAudiofeedback function)

commit 7518594440
Merge: e7a9c2b 7af128d
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 21:26:18 2026 +0800

    Merge remote-tracking branch 'origin/au-au' into au-au

    # Conflicts:
    #	app/src/main/java/com/example/jnicpp/bowling/BowlingCameraActivity.kt
    #	app/src/main/res/layout-land/activity_bowling_camera.xml
    #	app/src/main/res/layout/activity_bowling_camera.xml

commit e7a9c2b140
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 21:22:11 2026 +0800

    Add live per-step form feedback based on joint angles

    Extends pose angle tracking with knee bend (hip-knee-ankle), then uses
    it alongside the existing elbow/shoulder angles in a new
    PoseStageAdvisor to give a short live cue for whichever step of the
    approach is in progress: push-away on step 2, downswing on step 3,
    backswing on step 4, and knee-bend/arm-extension on the final step.
    Thresholds are starting defaults, not measured coaching data, and are
    expected to be retuned against real approach footage.

    Also fixes a layout bug found while testing this live: the new
    feedback text was chained via ConstraintLayout's toBottomOf to the
    step banner above it, so whenever that banner was hidden (GONE) the
    feedback text rendered at its collapsed zero-height position instead
    of staying put, landing on top of the recording indicator. Both now
    sit in a plain vertical LinearLayout, which collapses GONE children
    correctly.

    Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

commit 7af128d575
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 20:58:03 2026 +0800

    steps

    identify the steps

commit 2bb7c6ca7b
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 20:58:03 2026 +0800

    Add live final-position feedback and fix step detection accuracy

    Shows a live banner as each step of the approach is counted, ending in
    "FINAL POSITION - RELEASE!" once the 5th footfall lands. Also fixes
    LiveStepDetector's peak-prominence check, which compared a candidate
    footfall only to its immediate neighboring frame and silently dropped
    real steps that landed across several closely-spaced frames; it now
    tracks the true rise/fall trough on each side instead. Switches
    PoseAnalyzer to ML Kit's faster base pose model (the accurate model was
    starving the peak detector of frames on unaccelerated hardware), and
    makes LiveStepDetector's stillness-based reset toggleable, currently
    off for easier testing against recorded reference clips.

    Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-12 21:43:10 +08:00

250 lines
11 KiB
Kotlin

/**
* @file CameraViewModel.kt
* @brief UI/recording state machine and pose-data buffering for the bowling camera screen.
*/
package com.example.jnicpp.bowling
import android.app.Application
import androidx.lifecycle.AndroidViewModel
import androidx.lifecycle.viewModelScope
import kotlinx.coroutines.Job
import kotlinx.coroutines.delay
import kotlinx.coroutines.flow.MutableSharedFlow
import kotlinx.coroutines.flow.MutableStateFlow
import kotlinx.coroutines.flow.SharedFlow
import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.asSharedFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.isActive
import kotlinx.coroutines.launch
import kotlin.time.Duration.Companion.seconds
/**
* @brief Holds camera/recording UI state so it survives configuration
* changes and so the state machine lives outside the Activity.
*
* This class knows nothing about CameraX or ML Kit APIs directly --
* [BowlingCameraActivity] and [CameraXController] report events into it,
* and the UI observes it back out. That keeps this class trivially
* unit-testable (no Android camera framework involved).
*
* Extends [AndroidViewModel] rather than a plain ViewModel solely to reach
* an Application [android.content.Context] for [DetectorSettings.load] --
* see [onRecordingStarting].
*/
class CameraViewModel(application: Application) : AndroidViewModel(application) {
/** @brief The camera screen's overall recording state. */
sealed interface RecordingState {
/** @brief No recording in progress or starting. */
data object Idle : RecordingState
/** @brief A recording has been requested but hasn't started writing yet. */
data object Starting : RecordingState
/** @brief A recording is actively being written. @param elapsedSeconds Seconds elapsed since recording started. */
data class Recording(val elapsedSeconds: Long) : RecordingState
}
private val _recordingState = MutableStateFlow<RecordingState>(RecordingState.Idle)
/** @brief Current recording state, observed by the UI to drive button/timer/indicator visibility. */
val recordingState: StateFlow<RecordingState> = _recordingState.asStateFlow()
// Whether live pose detection/overlay is on. Only meant to change while
// recordingState is Idle -- the UI disables the toggle otherwise (see
// BowlingCameraActivity#renderRecordingState) since the recording
// pipeline picks its pose mode once at start.
private val _poseEnabled = MutableStateFlow(value = false)
/** @brief Whether pose detection/overlay is currently enabled. */
val poseEnabled: StateFlow<Boolean> = _poseEnabled.asStateFlow()
// Latest per-frame joint angles, for the overlay's angle-label text now
// and for frame-by-frame swing analysis/logging later. Null whenever
// there's no current pose result to derive them from (pose off, no
// frame processed yet, or a frame with no landmarks that met the
// confidence bar -- see PoseAngleCalculator).
private val _poseAngles = MutableStateFlow<PoseAngles?>(null)
/** @brief Joint angles computed for the most recent analyzed frame, or null if none available. */
@Suppress("unused")
val poseAngles: StateFlow<PoseAngles?> = _poseAngles.asStateFlow()
// Pose-frame buffering and live step counting for the current/most
// recent recording session -- see StepCountingSession's class doc for
// why this lives in its own class rather than inline here. Frames only
// flow into it while actually recording (see onPoseFrameUpdated) -- a
// live preview with pose overlay on but not recording doesn't feed it.
private val stepCountingSession = StepCountingSession()
/** @brief Time-ordered pose samples buffered for the current/most recent recording session. */
val poseFrames: List<PoseFrame> get() = stepCountingSession.poseFrames
// Live delivery-phase classification (starting stance, approach, etc)
private val posePhaseDetector = PosePhaseDetector()
private val _posePhase = MutableStateFlow<BowlingPhase?>(null)
//The bowler's current delivery phase, or null if no phase currently validates.
val posePhase: StateFlow<BowlingPhase?> = _posePhase.asStateFlow()
// Raw torso/knee/elbow angle readings behind posePhase above, for
// showing the bowler the actual numbers rather than just a pass/fail
// signal -- see PosePhaseDetector.Metrics.
private val _poseMetrics = MutableStateFlow<PosePhaseDetector.Metrics?>(null)
//This frame's torso/knee/elbow angle readings, or null if pose detection is off.
val poseMetrics: StateFlow<PosePhaseDetector.Metrics?> = _poseMetrics.asStateFlow()
/** @brief Steps detected so far in the current attempt, since the last reset. */
val stepEvents: StateFlow<List<StepEvent>> get() = stepCountingSession.stepEvents
// Live "how's my form right now" cue for whichever step is currently in
// progress -- see PoseStageAdvisor. Recomputed every frame alongside
// stepEvents so it's always tied to the same step count the UI already
// shows, and cleared on the same resets stepEvents is.
private val _poseStageFeedback = MutableStateFlow<String?>(null)
/** @brief Live form feedback for the current step, or null if there's nothing to say yet. */
val poseStageFeedback: StateFlow<String?> = _poseStageFeedback.asStateFlow()
private val _permissionsGranted = MutableStateFlow(value = false)
/** @brief Whether all required camera/microphone/storage permissions are currently granted. */
@Suppress("unused")
val permissionsGranted: StateFlow<Boolean> = _permissionsGranted.asStateFlow()
// One-shot user-facing error messages (camera unavailable, detector
// failure, storage failure, ...). SharedFlow, not StateFlow, so the same
// error doesn't get replayed and re-shown after a config change.
private val _errorEvents = MutableSharedFlow<String>(extraBufferCapacity = 4)
/** @brief One-shot user-facing error messages, e.g. for a Toast/Snackbar. */
val errorEvents: SharedFlow<String> = _errorEvents.asSharedFlow()
private var timerJob: Job? = null
/**
* @brief Records the result of a runtime permission request.
* @param granted true if all required permissions were granted.
*/
fun onPermissionsResult(granted: Boolean) {
_permissionsGranted.value = granted
}
/**
* @brief Toggles live pose detection/overlay on or off.
* @param enabled true to enable pose detection/overlay, false to disable.
*/
fun onPoseToggled(enabled: Boolean) {
_poseEnabled.value = enabled
if (!enabled) {
_poseAngles.value = null
posePhaseDetector.reset()
_posePhase.value = null
_poseMetrics.value = null
}
}
/**
* @brief Reports one analyzed frame's landmarks and joint angles.
*
* Always updates the live angle overlay. Buffering into [poseFrames]
* and live step counting are scoped to an actual recording (see
* [poseFrames]'s doc), so a preview with pose overlay on but not
* recording doesn't silently accumulate frames outside any session.
*
* @param landmarks EMA-smoothed landmarks for this frame, keyed by ML Kit's `PoseLandmark` type constant.
* @param angles Joint angles computed for this same frame.
*/
fun onPoseFrameUpdated(landmarks: Map<Int, SmoothedLandmark>, angles: PoseAngles) {
_poseAngles.value = angles
// Update pose phase detector with this frame's landmarks and angles (independent of step counter)
val phaseResult = posePhaseDetector.update(landmarks, angles)
_posePhase.value = phaseResult.phase
_poseMetrics.value = phaseResult.metrics
// Check if the current pose matches the Starting Stance (pure posture query)
val isStartingStance = (phaseResult.phase == BowlingPhase.STARTING_STANCE)
|| posePhaseDetector.isStartingStanceValid(phaseResult.metrics)
if (_recordingState.value is RecordingState.Recording) {
stepCountingSession.onFrame(
landmarks = landmarks,
angles = angles,
timestampMs = System.currentTimeMillis(),
isStartingPosition = isStartingStance,
)
val currentStepCount = stepEvents.value.size
_poseStageFeedback.value = PoseStageAdvisor.feedback(
stepNumber = currentStepCount.takeIf { it > 0 },
angles = angles,
)
}
}
/** @brief Manually resets step counting state for the current recording session. */
fun resetStepCounter() {
stepCountingSession.resetStepCounter()
_poseStageFeedback.value = null
}
/**
* @brief Marks a recording as being requested and resets all
* per-session buffering/detection state.
*
* Reloads [DetectorSettings] fresh here (rather than once at
* construction) so a tuning change made in [ParameterEditorActivity]
* takes effect on the very next recording, without needing to
* restart this screen.
*/
fun onRecordingStarting() {
_recordingState.value = RecordingState.Starting
stepCountingSession.startNewSession(DetectorSettings.load(getApplication()))
/*poseFrameBuffer.clear()
ankleHipSmoother.reset()
liveStepDetector.reset()
_stepEvents.value = emptyList()
_poseStageFeedback.value = null*/
}
/** @brief Marks a recording as actively writing and starts the elapsed-time timer. */
fun onRecordingStarted() {
timerJob?.cancel()
timerJob = viewModelScope.launch {
var seconds = 0L
while (isActive) {
_recordingState.value = RecordingState.Recording(seconds)
delay(1.seconds)
seconds++
}
}
}
/**
* @brief Marks the current recording as finished and stops the elapsed-time timer.
*
* [stepEvents] is left as whatever [LiveStepDetector] had already
* counted live -- it already reflects the last in-progress attempt's
* steps, and a fresh [StepDetector.detect] batch pass over the whole
* buffer here would ignore any mid-recording resets and overcount
* across separate attempts.
*/
fun onRecordingStopped() {
timerJob?.cancel()
timerJob = null
_recordingState.value = RecordingState.Idle
}
/**
* @brief Surfaces a one-shot error message to the UI, and clears a
* stuck recording indicator if one was in progress.
*
* A failed start/stop shouldn't leave the UI stuck showing a recording
* indicator that no longer reflects reality.
*
* @param message Human-readable error message to surface.
*/
fun postError(message: String) {
_errorEvents.tryEmit(message)
if (_recordingState.value != RecordingState.Idle) {
onRecordingStopped()
}
}
/** @brief Cancels the elapsed-time timer when this ViewModel is destroyed. */
override fun onCleared() {
timerJob?.cancel()
}
}