unplayable01/PinPoint
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit f0b9b2af67
Merge: 373d65a bbaa7bb
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Sat Sep 12 18:17:04 2026 +0800

    Merge branch 'merge-testing'

commit 373d65a9d5
Merge: 28fc652 dd0e6fd
Author: harine <harinesumen@gmail.com>
Date:   Sat Sep 12 17:34:27 2026 +0800

    Merge branch 'master' into Harine

commit 28fc652097
Author: harine <harinesumen@gmail.com>
Date:   Sat Sep 12 17:34:07 2026 +0800

    new stuff added yes

commit bbaa7bb04f
Merge: 1c6ced2 610bfd7
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 11:18:35 2026 +0800

    Merge branch 'Gabriel' into merge-testing

commit 1c6ced2dce
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 11:12:30 2026 +0800

    this good

commit dd0e6fdc39
Author: DefiantWanderer <yongwei1349@gmail.com>
Date:   Fri Sep 11 10:32:34 2026 +0800

    Revert "fixing compatibility issues"

    This reverts commit 564e7d83f8.

commit 564e7d83f8
Author: DefiantWanderer <yongwei1349@gmail.com>
Date:   Fri Sep 11 10:28:40 2026 +0800

    fixing compatibility issues

commit 610bfd7991
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 10:13:25 2026 +0800

    Merged Khalil

commit cfe76da1d3
Merge: 584818b 9fcdc13
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 10:03:51 2026 +0800

    Merge branch 'Khalil' into Gabriel

commit 584818b95a
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 12:47:06 2026 +0800

    added back phase toggle button

commit 17fad430c1
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 12:09:46 2026 +0800

    Cleaned up merge UI, added reset step button

commit ac4366d04d
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 00:31:47 2026 +0800

    Added some UI to recording

commit d601efcf43
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 00:13:06 2026 +0800

    Corrected 5 step approach phases & added placeholders for future logic

commit 7cf217488a
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Wed Sep 9 22:48:06 2026 +0800

    Cleared warnings

commit 981071ef74
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Wed Sep 9 20:09:18 2026 +0800

    Fix Merge, cleaned up UI

commit f30bbe2a34
Merge: 2f45115 7518594
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Tue Sep 8 14:11:21 2026 +0800

    Merge branch 'au-au' into Gabriel

commit 9fcdc13699
Author: Khalil Belabadia <belabakhalil@gmail.com>
Date:   Mon Sep 7 22:16:00 2026 +0800

    Audio feedback update

    Added the following files :

    - AudioCue.kt
    - AudioFeedbackEngine
    - AudioFeedbackSettings

    Edited files :

    - BowlingCameraActivity (added the triggerAudiofeedback function)

commit 7518594440
Merge: e7a9c2b 7af128d
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 21:26:18 2026 +0800

    Merge remote-tracking branch 'origin/au-au' into au-au

    # Conflicts:
    #	app/src/main/java/com/example/jnicpp/bowling/BowlingCameraActivity.kt
    #	app/src/main/res/layout-land/activity_bowling_camera.xml
    #	app/src/main/res/layout/activity_bowling_camera.xml

commit e7a9c2b140
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 21:22:11 2026 +0800

    Add live per-step form feedback based on joint angles

    Extends pose angle tracking with knee bend (hip-knee-ankle), then uses
    it alongside the existing elbow/shoulder angles in a new
    PoseStageAdvisor to give a short live cue for whichever step of the
    approach is in progress: push-away on step 2, downswing on step 3,
    backswing on step 4, and knee-bend/arm-extension on the final step.
    Thresholds are starting defaults, not measured coaching data, and are
    expected to be retuned against real approach footage.

    Also fixes a layout bug found while testing this live: the new
    feedback text was chained via ConstraintLayout's toBottomOf to the
    step banner above it, so whenever that banner was hidden (GONE) the
    feedback text rendered at its collapsed zero-height position instead
    of staying put, landing on top of the recording indicator. Both now
    sit in a plain vertical LinearLayout, which collapses GONE children
    correctly.

    Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

commit 7af128d575
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 20:58:03 2026 +0800

    steps

    identify the steps

commit 2bb7c6ca7b
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 20:58:03 2026 +0800

    Add live final-position feedback and fix step detection accuracy

    Shows a live banner as each step of the approach is counted, ending in
    "FINAL POSITION - RELEASE!" once the 5th footfall lands. Also fixes
    LiveStepDetector's peak-prominence check, which compared a candidate
    footfall only to its immediate neighboring frame and silently dropped
    real steps that landed across several closely-spaced frames; it now
    tracks the true rise/fall trough on each side instead. Switches
    PoseAnalyzer to ML Kit's faster base pose model (the accurate model was
    starving the peak detector of frames on unaccelerated hardware), and
    makes LiveStepDetector's stillness-based reset toggleable, currently
    off for easier testing against recorded reference clips.

    Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
jingwen121
2026-09-12 21:43:10 +08:00
parent b907747755
commit c6501d00f2
44 changed files with 1757 additions and 1063 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/src/main/java/com/example/jnicpp/bowling/BowlingCameraActivity.kt
+153 -45
View File
@@ -15,10 +15,10 @@ import androidx.activity.result.contract.ActivityResultContracts
import androidx.activity.viewModels
import androidx.appcompat.app.AppCompatActivity
import androidx.camera.core.CameraSelector
import androidx.core.content.ContextCompat
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.lifecycleScope
import androidx.lifecycle.repeatOnLifecycle
import androidx.core.content.ContextCompat
import com.example.jnicpp.R
import com.example.jnicpp.databinding.ActivityBowlingCameraBinding
import com.google.mlkit.vision.pose.PoseLandmark
@@ -45,6 +45,14 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
companion object {
private const val TAG = "BowlingCameraActivity"
// This app is built around a 5-step approach: the bowler is in
// their "final position" (planted/sliding, about to swing through
// and release) the moment the 5th foot-plant of the current attempt
// is detected. Step counting itself is LiveStepDetector's job (via
// CameraViewModel.stepEvents) -- this just interprets that count for
// the live banner below.
private const val FINAL_STEP_COUNT = 5
}
private lateinit var binding: ActivityBowlingCameraBinding
@@ -54,6 +62,16 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
private lateinit var cameraXController: CameraXController
private var lensFacing = CameraSelector.LENS_FACING_BACK
private lateinit var audioFeedbackSettings: AudioFeedbackSettings
private lateinit var audioFeedbackEngine: AudioFeedbackEngine
// Minimum gap between two audio cues, in milliseconds. Prevents the same
// coaching note from re-triggering the moment TTS finishes if the posture
// issue persists across many frames. DISCARD_IF_BUSY handles in-flight
// overlap; this cooldown handles the gap immediately after TTS goes silent.
private var lastCueMs = 0L
private val cueCooldownMs = 3_000L
// Throttled diagnostic for step-count troubleshooting: confirms whether
// ankles are actually clearing PoseSkeletonRenderer.MIN_LIKELIHOOD, since
// LiveStepDetector silently sees nothing for a foot until they do.
@@ -64,22 +82,22 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
// doc. Open only while a recording is in progress.
private lateinit var debugSessionLogger: DebugSessionLogger
// Rendering for the step-counter card and hold-to-reset indicator --
// Rendering for the step-counter card --
// see StepCounterUiController's class doc for why this isn't just
// inline here.
private lateinit var stepCounterUi: StepCounterUiController
// class for FeedbackUI
private lateinit var feedbackUI: FeedbackUI
private val stepLabels = listOf<Int>(
R.string.pose_phase_waiting,
private val stepLabels = listOf(
R.string.pose_phase_starting_stance,
R.string.first_step,
R.string.second_step,
R.string.third_step,
R.string.fourth_step,
R.string.end_position
R.string.pose_phase_approach,
R.string.pose_phase_pushaway,
R.string.pose_phase_back_swing,
R.string.pose_phase_power_step,
R.string.pose_phase_slide_and_release,
)
private var currentStepIndex = 0
private var currentPhaseToggleIndex = 0
private val permissionLauncher =
registerForActivityResult(ActivityResultContracts.RequestMultiplePermissions()) { _: Map<String, Boolean> ->
@@ -105,17 +123,31 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
setContentView(binding.root)
cameraExecutor = Executors.newSingleThreadExecutor()
cameraXController = CameraXController(applicationContext, cameraExecutor)
cameraXController = CameraXController(applicationContext, cameraExecutor).apply {
recordingOverlayStateProvider = {
CameraXController.OverlayState(
stepCount = viewModel.stepEvents.value.size,
phase = viewModel.posePhase.value,
stageFeedback = viewModel.poseStageFeedback.value,
metrics = viewModel.poseMetrics.value,
)
}
}
debugSessionLogger = DebugSessionLogger(applicationContext)
stepCounterUi = StepCounterUiController(
context = this,
cardStepCounter = binding.cardStepCounter,
textStepCountBig = binding.textStepCountBig,
layoutResetHint = binding.layoutResetHint,
progressHandRaise = binding.progressHandRaise,
textResetHint = binding.textResetHint
)
feedbackUI = FeedbackUI(this, binding.root)
feedbackUI = FeedbackUI(binding.root)
audioFeedbackSettings = AudioFeedbackSettings(applicationContext)
audioFeedbackEngine = AudioFeedbackEngine(applicationContext, audioFeedbackSettings)
// Reflect persisted mute state onto the switch before attaching the
// listener so the initial setChecked doesn't trigger the callback.
binding.switchAudio.isChecked = !audioFeedbackSettings.isMuted
binding.switchAudio.setOnCheckedChangeListener { _, isChecked ->
audioFeedbackSettings.isMuted = !isChecked
}
binding.poseOverlay.attachFeedback(feedbackUI)
binding.btnGrantPermissions.setOnClickListener {
@@ -130,7 +162,8 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
startActivity(Intent(this, ParameterEditorActivity::class.java))
}
}
binding.btnShowStep.setOnClickListener { onStepIncrease() }
binding.btnResetCounter.setOnClickListener { viewModel.resetStepCounter() }
binding.btnShowStep.setOnClickListener { onPhaseToggleClicked() }
observeViewModel()
@@ -151,7 +184,7 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
previewView = binding.cameraPreview,
callback = this,
lensFacing = lensFacing,
feedbackUi = feedbackUI
feedbackUi = feedbackUI,
)
}
@@ -230,13 +263,42 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
if (events.isNotEmpty()) {
Log.d(TAG, "Step ${events.size}: ${events.last()}")
}
// stepEvents is cleared back to empty on every reset
// (new recording, or LiveStepDetector seeing the
// bowler return to a stationary stance -- see
// CameraViewModel.onPoseFrameUpdated), so this banner
// naturally clears itself for the next attempt too.
//
// Shows every step as it's counted (not just the
// final one) so it's obvious on screen whether
// detection is actually seeing each footfall while
// testing/tuning it, rather than only finding out at
// step 5 that earlier steps were silently missed.
val isRecording = viewModel.recordingState.value is CameraViewModel.RecordingState.Recording
if (events.isEmpty() || !isRecording) {
binding.textFinalPosition.visibility = View.GONE
} else {
val reachedFinal = events.size >= FINAL_STEP_COUNT
binding.textFinalPosition.visibility = View.VISIBLE
binding.textFinalPosition.text = if (reachedFinal) {
getString(R.string.final_position_reached)
} else {
getString(R.string.step_reached_format, events.size)
}
binding.textFinalPosition.setTextColor(
ContextCompat.getColor(
this@BowlingCameraActivity,
if (reachedFinal) R.color.final_position_highlight else R.color.white,
),
)
}
}
}
launch {
viewModel.handRaiseProgress.collect { progress ->
stepCounterUi.renderHandRaiseProgress(
progress
)
viewModel.poseStageFeedback.collect { feedback ->
val isRecording = viewModel.recordingState.value is CameraViewModel.RecordingState.Recording
binding.textPoseStageFeedback.text = feedback
binding.textPoseStageFeedback.visibility = if ((feedback != null && isRecording)) View.VISIBLE else View.GONE
}
}
// Deliberately its own collector, independent of stepEvents
@@ -284,7 +346,9 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
when (state) {
is CameraViewModel.RecordingState.Idle -> {
binding.layoutRecordingIndicator.visibility = View.GONE
stepCounterUi.setVisible(false)
stepCounterUi.setVisible(visible = false)
binding.textFinalPosition.visibility = View.GONE
binding.textPoseStageFeedback.visibility = View.GONE
binding.btnRecord.isEnabled = true
binding.btnRecord.setText(R.string.record)
// Pose mode can only be changed between recordings, not
@@ -295,8 +359,6 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
// (see ParameterEditorActivity's class doc), so only offer
// it while there isn't one already in progress.
binding.btnEditor.visibility = View.VISIBLE
// Feedback UI - buttons only shown when recording
binding.btnShowStep.isEnabled = false
}
is CameraViewModel.RecordingState.Starting -> {
// Can't stop a recording that hasn't started yet, and pose
@@ -304,20 +366,17 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
binding.btnRecord.isEnabled = false
binding.switchPose.isEnabled = false
binding.btnEditor.visibility = View.GONE
binding.btnShowStep.isEnabled = true
binding.btnShowStep.setText(R.string.pose_phase_waiting)
}
is CameraViewModel.RecordingState.Recording -> {
binding.btnRecord.isEnabled = true
binding.btnRecord.setText(R.string.stop_recording)
binding.switchPose.isEnabled = false
binding.layoutRecordingIndicator.visibility = View.VISIBLE
stepCounterUi.setVisible(true)
stepCounterUi.setVisible(visible = true)
binding.btnEditor.visibility = View.GONE
val minutes = state.elapsedSeconds / 60
val seconds = state.elapsedSeconds % 60
binding.textTimer.text = String.format(Locale.US, "%02d:%02d", minutes, seconds)
binding.btnShowStep.isEnabled = true
}
}
}
@@ -358,6 +417,18 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
binding.textPoseFeedback.text = getString(R.string.pose_phase_pushaway)
binding.textPoseFeedback.setBackgroundColor(ContextCompat.getColor(this, R.color.Pushaway_ready))
}
BowlingPhase.BACK_SWING -> {
binding.textPoseFeedback.text = getString(R.string.pose_phase_back_swing)
binding.textPoseFeedback.setBackgroundColor(ContextCompat.getColor(this, R.color.Back_swing_ready))
}
BowlingPhase.POWER_STEP -> {
binding.textPoseFeedback.text = getString(R.string.pose_phase_power_step)
binding.textPoseFeedback.setBackgroundColor(ContextCompat.getColor(this, R.color.Power_step_ready))
}
BowlingPhase.SLIDE_AND_RELEASE -> {
binding.textPoseFeedback.text = getString(R.string.pose_phase_slide_and_release)
binding.textPoseFeedback.setBackgroundColor(ContextCompat.getColor(this, R.color.Slide_and_release_ready))
}
else -> {
binding.textPoseFeedback.text = getString(R.string.pose_phase_waiting)
binding.textPoseFeedback.setBackgroundColor(ContextCompat.getColor(this, R.color.Starting_stance_waiting))
@@ -381,7 +452,7 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
angleText(metrics.leftKneeAngleDegrees),
angleText(metrics.rightKneeAngleDegrees),
angleText(metrics.leftElbowAngleDegrees),
angleText(metrics.rightElbowAngleDegrees)
angleText(metrics.rightElbowAngleDegrees),
)
}
@@ -406,7 +477,7 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
private fun showPermissionRationale(showAsDenied: Boolean) {
binding.layoutPermissionRationale.visibility = View.VISIBLE
binding.textPermissionMessage.setText(
if (showAsDenied) R.string.permission_denied_message else R.string.permission_rationale_message
if (showAsDenied) R.string.permission_denied_message else R.string.permission_rationale_message,
)
}
@@ -439,7 +510,7 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
Toast.makeText(
this,
"${outputUri.lastPathSegment ?: outputUri.toString()} (debug trace saved to Downloads/bowling)",
Toast.LENGTH_LONG
Toast.LENGTH_LONG,
).show()
}
@@ -476,16 +547,16 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
override fun onPoseResult(result: PoseAnalyzer.PoseFrameResult) {
binding.poseOverlay.update(result)
viewModel.onPoseFrameUpdated(result.landmarks, result.angles)
triggerAudioFeedback(result.angles)
val frameTimestampMs = System.currentTimeMillis()
debugSessionLogger.log(
result.landmarks,
frameTimestampMs,
viewModel.stepEvents.value.size,
viewModel.handRaiseProgress.value
)
if (frameTimestampMs - lastLandmarkLogMs >= 1000) {
if ((frameTimestampMs - lastLandmarkLogMs) >= 1000) {
lastLandmarkLogMs = frameTimestampMs
val leftAnkle = result.landmarks[PoseLandmark.LEFT_ANKLE]
val rightAnkle = result.landmarks[PoseLandmark.RIGHT_ANKLE]
@@ -498,30 +569,67 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
"Likelihood (need >= ${PoseSkeletonRenderer.MIN_LIKELIHOOD}) -- " +
"ankle L=${leftAnkle?.inFrameLikelihood} R=${rightAnkle?.inFrameLikelihood}, " +
"hip L=${leftHip?.inFrameLikelihood} R=${rightHip?.inFrameLikelihood}, " +
"shoulder L=${leftShoulder?.inFrameLikelihood} R=${rightShoulder?.inFrameLikelihood}"
"shoulder L=${leftShoulder?.inFrameLikelihood} R=${rightShoulder?.inFrameLikelihood}",
)
}
}
/**
* @brief Evaluates [angles] against basic form thresholds and speaks a
* coaching cue when a limit is exceeded.
*
* Only one cue is emitted per call (the highest-priority issue found
* first). A [cueCooldownMs] guard prevents the same note from
* re-firing the instant TTS goes silent after speaking it.
* [QueuePolicy.DISCARD_IF_BUSY] means any cue arriving while TTS is
* mid-sentence is silently dropped at the engine level, so live
* per-frame calls here never stack up.
*
* Angle thresholds below are starting-point estimates; they should be
* calibrated against recorded sessions once the posture-analysis
* milestone establishes target ranges per bowling phase.
*
* @param angles Joint angles computed for the current frame.
*/
private fun triggerAudioFeedback(angles: PoseAngles) {
val now = System.currentTimeMillis()
if (now - lastCueMs < cueCooldownMs) return
// Prefer the dominant (right) arm; fall back to left if right is
// not detected. Null means neither arm was reliably seen this frame.
val elbowAngle = angles.rightElbow ?: angles.leftElbow
val shoulderAngle = angles.rightShoulder ?: angles.leftShoulder
val cue: AudioCue? = when {
// Arm locked straight well before the release point.
elbowAngle != null && elbowAngle > 160f -> AudioCue.BendElbow
// Arm over-bent; disrupts swing plane and release.
elbowAngle != null && elbowAngle < 80f -> AudioCue.StraightenArm
// Bowling-side shoulder lifting during the swing.
shoulderAngle != null && shoulderAngle > 140f -> AudioCue.LowerShoulder
else -> null
}
if (cue != null) {
audioFeedbackEngine.speak(cue, QueuePolicy.DISCARD_IF_BUSY)
lastCueMs = now
}
}
/** @brief Releases the camera controller, shuts down the analysis executor, and closes any open debug trace file. */
override fun onDestroy() {
super.onDestroy()
cameraXController.release()
cameraExecutor.shutdown()
debugSessionLogger.stop()
audioFeedbackEngine.release()
}
/**
* @brief Advances the step index and updates the step button label.
*
* This method increments the current step index, cycling back to zero
* once the end of the [stepLabels] list is reached. It then updates the
* `btnShowStep` text to reflect the new step, ensuring the UI button
* always displays the correct label for the current position in the
* sequence.
* @brief Cycles through the delivery phase labels on the manual toggle button.
*/
fun onStepIncrease() {
currentStepIndex = (currentStepIndex + 1) % stepLabels.size
binding.btnShowStep.setText(stepLabels[currentStepIndex])
private fun onPhaseToggleClicked() {
currentPhaseToggleIndex = (currentPhaseToggleIndex + 1) % stepLabels.size
binding.btnShowStep.setText(stepLabels[currentPhaseToggleIndex])
}
}