Compare commits
7
Commits
merge-testing
...
Harine
| Author | SHA1 | Date | |
|---|---|---|---|
 harineandClaude Sonnet 5
|
aa25d897dc | ||
|
harine
|
ac3eb98b9c | ||
|
Gabriel Low
|
f0b9b2af67 | ||
|
harine
|
373d65a9d5 | ||
|
harine
|
28fc652097 | ||
 DefiantWanderer
|
dd0e6fdc39 | ||
|
DefiantWanderer
|
564e7d83f8 |
@@ -0,0 +1,37 @@
|
||||
# Git LFS tracking for BowlEye
|
||||
# Binary/large-asset types are stored via LFS and lockable, since binaries
|
||||
# can't be merged. Run `git lfs lock <file>` before editing one of these.
|
||||
# See docs/ and the "So you got your VM" onboarding deck for context.
|
||||
|
||||
# ML models (pose detection, custom-trained)
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text lockable
|
||||
|
||||
# Video / audio (bowling capture footage, test clips)
|
||||
*.mp4 filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.mov filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.wav filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.mp3 filter=lfs diff=lfs merge=lfs -text lockable
|
||||
|
||||
# Images
|
||||
*.png filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.jpg filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.jpeg filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.webp filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.psd filter=lfs diff=lfs merge=lfs -text lockable
|
||||
|
||||
# Android build / signing artifacts
|
||||
*.apk filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.aab filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.jar filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.aar filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.so filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.keystore filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.jks filter=lfs diff=lfs merge=lfs -text lockable
|
||||
|
||||
# Misc large/binary
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.fbx filter=lfs diff=lfs merge=lfs -text lockable
|
||||
*.blend filter=lfs diff=lfs merge=lfs -text lockable
|
||||
+30
-15
@@ -1,20 +1,35 @@
|
||||
*.iml
|
||||
.gradle
|
||||
/local.properties
|
||||
app/.idea/
|
||||
/.idea/caches
|
||||
/.idea/libraries
|
||||
/.idea/modules.xml
|
||||
/.idea/workspace.xml
|
||||
/.idea/navEditor.xml
|
||||
/.idea/assetWizardSettings.xml
|
||||
.DS_Store
|
||||
# Build output
|
||||
/build
|
||||
/captures
|
||||
.externalNativeBuild
|
||||
.cxx
|
||||
local.properties
|
||||
/.idea
|
||||
|
||||
gradle\wrapper\gradle-wrapper.properties
|
||||
gradle\libs.versions.toml
|
||||
# Gradle
|
||||
.gradle
|
||||
|
||||
# Local machine config (never commit — contains local SDK paths/secrets)
|
||||
local.properties
|
||||
/local.properties
|
||||
|
||||
# Android Studio / IntelliJ
|
||||
*.iml
|
||||
/.idea
|
||||
app/.idea/
|
||||
.idea/caches
|
||||
.idea/libraries
|
||||
.idea/modules.xml
|
||||
.idea/workspace.xml
|
||||
.idea/navEditor.xml
|
||||
.idea/assetWizardSettings.xml
|
||||
.idea/deploymentTargetSelector.xml
|
||||
.idea/deviceManager.xml
|
||||
|
||||
# Kotlin daemon cache
|
||||
.kotlin
|
||||
|
||||
# Visual Studio (native/CMake tooling)
|
||||
.vs/
|
||||
|
||||
# OS
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
Generated
-22
@@ -1,22 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="deploymentTargetSelector">
|
||||
<selectionStates>
|
||||
<SelectionState runConfigName="Unnamed">
|
||||
<option name="selectionMode" value="DROPDOWN" />
|
||||
<DialogSelection />
|
||||
</SelectionState>
|
||||
<SelectionState runConfigName="app">
|
||||
<option name="selectionMode" value="DROPDOWN" />
|
||||
<DropdownSelection timestamp="2026-09-05T10:57:53.251732600Z">
|
||||
<Target type="DEFAULT_BOOT">
|
||||
<template>
|
||||
<DeviceId pluginId="FirebaseDirectAccess" type="TEMPLATE" identifier="model_id=pa1qksx/36" />
|
||||
</template>
|
||||
</Target>
|
||||
</DropdownSelection>
|
||||
<DialogSelection />
|
||||
</SelectionState>
|
||||
</selectionStates>
|
||||
</component>
|
||||
</project>
|
||||
Generated
-13
@@ -1,13 +0,0 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project version="4">
|
||||
<component name="DeviceTable">
|
||||
<option name="columnSorters">
|
||||
<list>
|
||||
<ColumnSorterState>
|
||||
<option name="column" value="Name" />
|
||||
<option name="order" value="ASCENDING" />
|
||||
</ColumnSorterState>
|
||||
</list>
|
||||
</option>
|
||||
</component>
|
||||
</project>
|
||||
@@ -0,0 +1,84 @@
|
||||
# PinPoint (BowlEye)
|
||||
|
||||
An Android application for analyzing a bowler's approach and delivery form using on-device pose detection. The app captures video of a bowler via the phone camera, tracks body landmarks in real time, and surfaces step-count, joint-angle, and phase feedback to help improve technique.
|
||||
|
||||
See [`docs/`](docs/) for the project proposal, architecture, and deliverables/ownership breakdown.
|
||||
|
||||
## Tech stack
|
||||
|
||||
- **Language:** Kotlin (application/feature code), Java (legacy entry point), C++ (native game shell)
|
||||
- **Platform:** Android (min SDK 24, target/compile SDK 36)
|
||||
- **Build system:** Gradle (Kotlin DSL version catalog) + CMake 3.22.1 for the native module
|
||||
- **Key libraries:** CameraX (capture), ML Kit Pose Detection — accurate model (landmark tracking), Kotlin Coroutines, AndroidX Lifecycle/ViewModel
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Android Studio** (current stable channel) — bundles a compatible JDK, so no separate JDK install is required
|
||||
- **Android SDK** with:
|
||||
- Android SDK Platform 36 (and 36.1)
|
||||
- NDK (side by side) — a recent version compatible with AGP
|
||||
- CMake 3.22.1
|
||||
- Android SDK Build-Tools 36.0.0
|
||||
- A physical Android device with a camera (recommended) or an emulator with a virtual/webcam camera. Real camera input is strongly recommended since pose detection needs an actual moving subject.
|
||||
|
||||
> The NDK/CMake/platform components above do **not** need to be installed manually — both Android Studio's "install missing components" prompt and a plain `./gradlew` command-line build will fetch them automatically on first sync/build, provided you have internet access and accept the SDK license prompts.
|
||||
|
||||
## Getting started
|
||||
|
||||
### Option A — Android Studio
|
||||
|
||||
1. `File > Open` and select the repository root.
|
||||
2. Let Gradle sync; approve any "install missing SDK components" prompts.
|
||||
3. Connect a device (enable **Developer Options > USB debugging** on the phone) or start an emulator.
|
||||
4. Click **Run** ▶ with the `app` configuration selected.
|
||||
|
||||
### Option B — Command line
|
||||
|
||||
```bash
|
||||
# Windows
|
||||
.\gradlew.bat assembleDebug
|
||||
.\gradlew.bat installDebug
|
||||
|
||||
# macOS / Linux
|
||||
./gradlew assembleDebug
|
||||
./gradlew installDebug
|
||||
```
|
||||
|
||||
Then launch an activity directly, e.g.:
|
||||
|
||||
```bash
|
||||
adb shell am start -n com.example.jnicpp/.MainActivity
|
||||
adb shell am start -n com.example.jnicpp/.bowling.BowlingCameraActivity
|
||||
```
|
||||
|
||||
### First-time local setup notes
|
||||
|
||||
- Gradle needs a `local.properties` file **at the repository root** (not inside `app/`) pointing at your local Android SDK, e.g.:
|
||||
```
|
||||
sdk.dir=/Users/you/Library/Android/sdk
|
||||
```
|
||||
Android Studio creates/updates this automatically on first sync. This file is machine-specific and is gitignored — never commit it.
|
||||
- The app requests **Camera** and **Record Audio** permissions at runtime; grant both to use the bowling analysis screen.
|
||||
|
||||
## Project structure
|
||||
|
||||
```
|
||||
app/
|
||||
├── src/main/cpp/ # Native C++ game shell (OpenGL ES menu, JNI bridge), built via CMake
|
||||
├── src/main/java/.../ # MainActivity (native GL menu host)
|
||||
└── src/main/java/.../bowling/ # Bowling capture + pose analysis feature (Kotlin, CameraX + ML Kit)
|
||||
docs/ # Proposal, architecture, design, and deliverables documentation
|
||||
```
|
||||
|
||||
The native GL menu (`MainActivity`) and the bowling camera/pose-analysis screen (`BowlingCameraActivity`) are currently two independent entry points — see [`docs/architecture.md`](docs/architecture.md) for how they're intended to connect.
|
||||
|
||||
## Running tests
|
||||
|
||||
```bash
|
||||
./gradlew test # JVM unit tests
|
||||
./gradlew connectedCheck # Instrumented tests (requires a connected device/emulator)
|
||||
```
|
||||
|
||||
## Cross-platform notes
|
||||
|
||||
The native game-state code under `app/src/main/cpp/my_gl_app/` is written behind a `Platform.h` seam with both `PLATFORM_ANDROID` and `PLATFORM_WINDOWS` (GLFW) code paths, so the menu/state-machine logic itself is portable. Only the Android (Gradle/CMake) build is wired up today; there is no standalone desktop build target yet.
|
||||
@@ -89,13 +89,17 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
|
||||
// class for FeedbackUI
|
||||
private lateinit var feedbackUI: FeedbackUI
|
||||
|
||||
// The team's 5-step terminology, in order -- deliberately not one entry
|
||||
// per BowlingPhase, since some phases span two of these (see
|
||||
// PosePhaseDetector.phaseForStep).
|
||||
private val stepLabels = listOf(
|
||||
R.string.pose_phase_starting_stance,
|
||||
R.string.pose_phase_approach,
|
||||
R.string.step_term_half_step,
|
||||
R.string.step_term_preparation_step,
|
||||
R.string.pose_phase_pushaway,
|
||||
R.string.pose_phase_back_swing,
|
||||
R.string.pose_phase_power_step,
|
||||
R.string.pose_phase_slide_and_release,
|
||||
R.string.step_term_slide,
|
||||
R.string.step_term_finishing_position,
|
||||
)
|
||||
private var currentPhaseToggleIndex = 0
|
||||
|
||||
@@ -137,6 +141,7 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
|
||||
stepCounterUi = StepCounterUiController(
|
||||
cardStepCounter = binding.cardStepCounter,
|
||||
textStepCountBig = binding.textStepCountBig,
|
||||
textStepCounterLabel = binding.textStepCounterLabel,
|
||||
)
|
||||
feedbackUI = FeedbackUI(binding.root)
|
||||
audioFeedbackSettings = AudioFeedbackSettings(applicationContext)
|
||||
@@ -259,7 +264,7 @@ class BowlingCameraActivity : AppCompatActivity(), CameraXController.Callback {
|
||||
}
|
||||
launch {
|
||||
viewModel.stepEvents.collect { events ->
|
||||
stepCounterUi.renderStepCount(events.size)
|
||||
stepCounterUi.renderStepCount(events.size, events.lastOrNull()?.poseConfirmed ?: true)
|
||||
if (events.isNotEmpty()) {
|
||||
Log.d(TAG, "Step ${events.size}: ${events.last()}")
|
||||
}
|
||||
|
||||
@@ -164,6 +164,7 @@ class CameraViewModel(application: Application) : AndroidViewModel(application)
|
||||
angles = angles,
|
||||
timestampMs = System.currentTimeMillis(),
|
||||
isStartingPosition = isStartingStance,
|
||||
currentPhase = phaseResult.phase,
|
||||
)
|
||||
val currentStepCount = stepEvents.value.size
|
||||
_poseStageFeedback.value = PoseStageAdvisor.feedback(
|
||||
|
||||
@@ -399,14 +399,16 @@ class CameraXController(
|
||||
}
|
||||
|
||||
private fun drawPhaseBadgeOverlay(canvas: Canvas, phase: BowlingPhase, scale: Float) {
|
||||
val label = when (phase) {
|
||||
BowlingPhase.STARTING_STANCE -> "Starting Stance"
|
||||
BowlingPhase.APPROACH -> "Approach"
|
||||
BowlingPhase.PUSHAWAY -> "Pushaway"
|
||||
BowlingPhase.BACK_SWING -> "Backswing"
|
||||
BowlingPhase.POWER_STEP -> "Power Step"
|
||||
BowlingPhase.SLIDE_AND_RELEASE -> "Slide & Release"
|
||||
}
|
||||
val label = appContext.getString(
|
||||
when (phase) {
|
||||
BowlingPhase.STARTING_STANCE -> R.string.pose_phase_starting_stance
|
||||
BowlingPhase.APPROACH -> R.string.pose_phase_approach
|
||||
BowlingPhase.PUSHAWAY -> R.string.pose_phase_pushaway
|
||||
BowlingPhase.BACK_SWING -> R.string.pose_phase_back_swing
|
||||
BowlingPhase.POWER_STEP -> R.string.pose_phase_power_step
|
||||
BowlingPhase.SLIDE_AND_RELEASE -> R.string.pose_phase_slide_and_release
|
||||
}
|
||||
)
|
||||
val colorRes = when (phase) {
|
||||
BowlingPhase.STARTING_STANCE -> R.color.Starting_stance_ready
|
||||
BowlingPhase.APPROACH -> R.color.Approach_ready
|
||||
|
||||
@@ -303,32 +303,40 @@ class PosePhaseDetector(
|
||||
|
||||
companion object {
|
||||
/**
|
||||
* @brief Maps a 5-step approach step count (0..5) to its corresponding [BowlingPhase].
|
||||
* @brief Maps a 5-step approach step count (0..5) to the [BowlingPhase]
|
||||
* the bowler should be in once that step has landed.
|
||||
*
|
||||
* step 0 -> STARTING_STANCE
|
||||
* step 1 -> APPROACH
|
||||
* step 2 -> PUSHAWAY
|
||||
* step 3 -> BACK_SWING
|
||||
* step 4 -> POWER_STEP
|
||||
* step 5 -> SLIDE_AND_RELEASE
|
||||
* Follows the team's 5-step terminology: starting position -> 1/2 step
|
||||
* -> preparation step -> push away & backswing -> power step -> slide
|
||||
* -> finishing position. The 1/2 and preparation steps are both
|
||||
* [BowlingPhase.APPROACH] (ball still held, CG moving forward); the
|
||||
* ball is pushed away into the swing on step 3, reaches the peak of
|
||||
* the backswing as the very short power step lands on step 4, and is
|
||||
* released during the slide on step 5.
|
||||
*
|
||||
* step 0 -> STARTING_STANCE (starting position)
|
||||
* steps 1-2 -> APPROACH (1/2 step, preparation step)
|
||||
* step 3 -> PUSHAWAY (push away & backswing; BACK_SWING shares this step)
|
||||
* step 4 -> POWER_STEP
|
||||
* step 5+ -> SLIDE_AND_RELEASE (slide, finishing position)
|
||||
*/
|
||||
fun phaseForStep(stepCount: Int): BowlingPhase = when {
|
||||
stepCount <= 0 -> BowlingPhase.STARTING_STANCE
|
||||
stepCount == 1 -> BowlingPhase.APPROACH
|
||||
stepCount == 2 -> BowlingPhase.PUSHAWAY
|
||||
stepCount == 3 -> BowlingPhase.BACK_SWING
|
||||
stepCount <= 2 -> BowlingPhase.APPROACH
|
||||
stepCount == 3 -> BowlingPhase.PUSHAWAY
|
||||
stepCount == 4 -> BowlingPhase.POWER_STEP
|
||||
else -> BowlingPhase.SLIDE_AND_RELEASE
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief Maps a [BowlingPhase] to its corresponding 5-step approach step count.
|
||||
* @brief The first step of a 5-step approach at which [phase] begins
|
||||
* -- see [phaseForStep] for the full mapping. APPROACH spans
|
||||
* steps 1-2, and PUSHAWAY/BACK_SWING both begin on step 3.
|
||||
*/
|
||||
@Suppress("unused")
|
||||
fun stepForPhase(phase: BowlingPhase): Int = when (phase) {
|
||||
BowlingPhase.STARTING_STANCE -> 0
|
||||
BowlingPhase.APPROACH -> 1
|
||||
BowlingPhase.PUSHAWAY -> 2
|
||||
BowlingPhase.PUSHAWAY -> 3
|
||||
BowlingPhase.BACK_SWING -> 3
|
||||
BowlingPhase.POWER_STEP -> 4
|
||||
BowlingPhase.SLIDE_AND_RELEASE -> 5
|
||||
|
||||
@@ -31,19 +31,24 @@ package com.example.jnicpp.bowling
|
||||
*/
|
||||
object PoseStageAdvisor {
|
||||
|
||||
// Step-2 cue: ball still close to the body just after push-away, so the
|
||||
// swing-arm shoulder angle (elbow-shoulder-hip) should still be small.
|
||||
private const val PUSH_AWAY_MAX_SHOULDER_DEG = 30f
|
||||
// Step names follow the team's 5-step terminology (see
|
||||
// PosePhaseDetector.phaseForStep): 1/2 step, preparation step, push away
|
||||
// & backswing, power step, slide -> finishing position.
|
||||
|
||||
// Step-3 cue: arm swinging down and back past the body.
|
||||
private const val DOWNSWING_MIN_SHOULDER_DEG = 25f
|
||||
private const val DOWNSWING_MAX_SHOULDER_DEG = 75f
|
||||
// Preparation-step cue: the ball is still held close to the body before
|
||||
// the push away, so the swing-arm shoulder angle (elbow-shoulder-hip)
|
||||
// should still be small.
|
||||
private const val PREPARATION_MAX_SHOULDER_DEG = 30f
|
||||
|
||||
// Step-4 cue: arm swinging well back behind the body.
|
||||
private const val BACKSWING_MIN_SHOULDER_DEG = 60f
|
||||
// Push away & backswing cue: ball pushed out and swinging past the body.
|
||||
private const val PUSH_AWAY_MIN_SHOULDER_DEG = 25f
|
||||
private const val PUSH_AWAY_MAX_SHOULDER_DEG = 75f
|
||||
|
||||
// Final-position cues: front knee bent to lower the slide, swing arm
|
||||
// relatively straight through the release.
|
||||
// Power-step cue: ball at the peak of the backswing, well behind the body.
|
||||
private const val POWER_STEP_MIN_SHOULDER_DEG = 60f
|
||||
|
||||
// Slide / finishing-position cues: front knee bent to lower the slide,
|
||||
// swing arm relatively straight through the release.
|
||||
private const val RELEASE_MAX_KNEE_DEG = 140f
|
||||
private const val RELEASE_MIN_ELBOW_DEG = 150f
|
||||
|
||||
@@ -61,25 +66,35 @@ object PoseStageAdvisor {
|
||||
val frontKnee = smallerOf(angles.leftKnee, angles.rightKnee)
|
||||
|
||||
return when {
|
||||
(stepNumber == null || stepNumber <= 1) -> "Starting position - stay relaxed"
|
||||
(stepNumber == null || stepNumber <= 0) -> "Starting position - stay relaxed"
|
||||
|
||||
stepNumber == 1 -> "½ step - small step, weight shifting forward"
|
||||
|
||||
stepNumber == 2 -> swingShoulder?.let {
|
||||
if (it <= PUSH_AWAY_MAX_SHOULDER_DEG) "Good push-away" else "Push the ball out first"
|
||||
if (it <= PREPARATION_MAX_SHOULDER_DEG) {
|
||||
"Good preparation step"
|
||||
} else {
|
||||
"Keep the ball close until the push away"
|
||||
}
|
||||
}
|
||||
|
||||
stepNumber == 3 -> swingShoulder?.let {
|
||||
if (it in DOWNSWING_MIN_SHOULDER_DEG..DOWNSWING_MAX_SHOULDER_DEG) {
|
||||
"Good downswing"
|
||||
if (it in PUSH_AWAY_MIN_SHOULDER_DEG..PUSH_AWAY_MAX_SHOULDER_DEG) {
|
||||
"Good push away & backswing"
|
||||
} else {
|
||||
"Let the arm swing naturally"
|
||||
"Push the ball up and let it swing"
|
||||
}
|
||||
}
|
||||
|
||||
stepNumber == 4 -> swingShoulder?.let {
|
||||
if (it >= BACKSWING_MIN_SHOULDER_DEG) "Good backswing" else "Swing the arm further back"
|
||||
if (it >= POWER_STEP_MIN_SHOULDER_DEG) {
|
||||
"Good power step - ball at the top"
|
||||
} else {
|
||||
"Let the ball swing higher on the power step"
|
||||
}
|
||||
}
|
||||
|
||||
else -> { // final step (5+)
|
||||
else -> { // slide & finishing position (5+)
|
||||
val kneeGood = frontKnee != null && frontKnee <= RELEASE_MAX_KNEE_DEG
|
||||
val armGood = swingElbow != null && swingElbow >= RELEASE_MIN_ELBOW_DEG
|
||||
when {
|
||||
|
||||
@@ -6,16 +6,22 @@ package com.example.jnicpp.bowling
|
||||
|
||||
import android.view.View
|
||||
import android.widget.TextView
|
||||
import androidx.core.content.ContextCompat
|
||||
import com.example.jnicpp.R
|
||||
|
||||
/**
|
||||
* @brief Owns rendering for [BowlingCameraActivity]'s step-counter card.
|
||||
*
|
||||
* @param cardStepCounter The step-counter card container view.
|
||||
* @param textStepCountBig The large step-count number TextView.
|
||||
* @param textStepCounterLabel The small "STEPS" label below the count,
|
||||
* repurposed to flag when the latest step wasn't corroborated by
|
||||
* [PosePhaseDetector] -- see [renderStepCount]'s `latestPoseConfirmed`.
|
||||
*/
|
||||
class StepCounterUiController(
|
||||
private val cardStepCounter: View,
|
||||
private val textStepCountBig: TextView,
|
||||
private val textStepCounterLabel: TextView,
|
||||
) {
|
||||
// Last step count rendered, so pulse() in renderStepCount only plays
|
||||
// when a new step actually pushed the count up.
|
||||
@@ -36,14 +42,34 @@ class StepCounterUiController(
|
||||
|
||||
/**
|
||||
* @brief Renders the current step count, pulsing the card if a new step was just confirmed.
|
||||
*
|
||||
* Never withholds or delays a step because of [latestPoseConfirmed] --
|
||||
* the ankle-peak count is always trusted (see [StepEvent.poseConfirmed]'s
|
||||
* doc for why); this only swaps the small label below the number to flag
|
||||
* a disagreement for whoever's testing/tuning detection to notice.
|
||||
*
|
||||
* @param stepCount Total steps counted so far in the current attempt.
|
||||
* @param latestPoseConfirmed Whether the most recently counted step (if
|
||||
* any) was corroborated by [PosePhaseDetector] at the time it was
|
||||
* counted; ignored when [stepCount] is 0.
|
||||
*/
|
||||
fun renderStepCount(stepCount: Int) {
|
||||
fun renderStepCount(stepCount: Int, latestPoseConfirmed: Boolean = true) {
|
||||
textStepCountBig.text = stepCount.toString()
|
||||
if (stepCount > lastRenderedStepCount) {
|
||||
pulse()
|
||||
}
|
||||
lastRenderedStepCount = stepCount
|
||||
|
||||
val flagged = stepCount > 0 && !latestPoseConfirmed
|
||||
textStepCounterLabel.text = textStepCounterLabel.context.getString(
|
||||
if (flagged) R.string.step_counter_label_unconfirmed else R.string.step_counter_label
|
||||
)
|
||||
textStepCounterLabel.setTextColor(
|
||||
ContextCompat.getColor(
|
||||
textStepCounterLabel.context,
|
||||
if (flagged) R.color.recording_red else R.color.step_counter_accent,
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
/** @brief Briefly scales the step counter up and back down, drawing the eye to a newly confirmed step. */
|
||||
|
||||
@@ -57,12 +57,17 @@ class StepCountingSession {
|
||||
* @param angles Joint angles computed for this same frame.
|
||||
* @param timestampMs Wall-clock time this frame was analyzed, in milliseconds.
|
||||
* @param isStartingPosition Whether the bowler is currently in the starting position.
|
||||
* @param currentPhase [PosePhaseDetector]'s currently-classified delivery
|
||||
* phase for this same frame, or null if none currently validates
|
||||
* -- used only to set each new [StepEvent.poseConfirmed] below,
|
||||
* never to gate step counting itself (see that field's doc).
|
||||
*/
|
||||
fun onFrame(
|
||||
landmarks: Map<Int, SmoothedLandmark>,
|
||||
angles: PoseAngles,
|
||||
timestampMs: Long,
|
||||
isStartingPosition: Boolean = false,
|
||||
currentPhase: BowlingPhase? = null,
|
||||
) {
|
||||
val smoothedAnkleHip = ankleHipSmoother.smooth(landmarks)
|
||||
val frame = buildPoseFrame(
|
||||
@@ -78,7 +83,15 @@ class StepCountingSession {
|
||||
_stepEvents.value = emptyList()
|
||||
}
|
||||
if (result.newSteps.isNotEmpty()) {
|
||||
_stepEvents.value += result.newSteps
|
||||
// Compared phase-to-phase (via each phase's starting step) rather
|
||||
// than against the raw step number, since one phase can span
|
||||
// several steps -- e.g. APPROACH covers steps 1-2, so a pose still
|
||||
// in APPROACH is correct when step 2 lands.
|
||||
val posePhaseStart = PosePhaseDetector.stepForPhase(currentPhase ?: BowlingPhase.STARTING_STANCE)
|
||||
_stepEvents.value += result.newSteps.map { step ->
|
||||
val expectedPhaseStart = PosePhaseDetector.stepForPhase(PosePhaseDetector.phaseForStep(step.stepIndex))
|
||||
step.copy(poseConfirmed = posePhaseStart >= expectedPhaseStart)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -19,11 +19,21 @@ enum class Foot { LEFT, RIGHT }
|
||||
* @param timestampMs Time this foot-plant was detected, in milliseconds.
|
||||
* @param foot Which foot planted.
|
||||
* @param stepIndex 1-based position of this step in the overall approach sequence.
|
||||
* @param poseConfirmed Whether [PosePhaseDetector]'s independently-classified
|
||||
* posture had already reached the delivery phase this step index
|
||||
* expects (see [PosePhaseDetector.phaseForStep]) at the moment this
|
||||
* step was counted -- see [StepCountingSession.onFrame]. The ankle-peak
|
||||
* count is trusted either way (this never blocks a step from being
|
||||
* counted); false just flags that the two signals disagreed, e.g. a
|
||||
* step 1 landing before pose ever confirmed the bowler had actually
|
||||
* left the starting stance. Always true from [StepDetector.detect]'s
|
||||
* batch pass, which has no phase information to compare against.
|
||||
*/
|
||||
data class StepEvent(
|
||||
val timestampMs: Long,
|
||||
val foot: Foot,
|
||||
val stepIndex: Int,
|
||||
val poseConfirmed: Boolean = true,
|
||||
)
|
||||
|
||||
/**
|
||||
|
||||
@@ -102,6 +102,7 @@
|
||||
android:textStyle="bold" />
|
||||
|
||||
<TextView
|
||||
android:id="@+id/text_step_counter_label"
|
||||
android:layout_width="wrap_content"
|
||||
android:layout_height="wrap_content"
|
||||
android:text="@string/step_counter_label"
|
||||
|
||||
@@ -142,6 +142,7 @@
|
||||
android:textStyle="bold" />
|
||||
|
||||
<TextView
|
||||
android:id="@+id/text_step_counter_label"
|
||||
android:layout_width="wrap_content"
|
||||
android:layout_height="wrap_content"
|
||||
android:text="@string/step_counter_label"
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
<string name="recording_timer_placeholder">00:00</string>
|
||||
<string name="step_count_big_placeholder">0</string>
|
||||
<string name="step_counter_label">STEPS</string>
|
||||
<string name="step_counter_label_unconfirmed">STEPS · UNCONFIRMED BY POSE</string>
|
||||
<string name="reset_counter">Reset Steps</string>
|
||||
<string name="reset_hint_idle">✋ Raise a hand, hold 5s to reset</string>
|
||||
<string name="reset_hint_holding">Keep holding… %1$d%%</string>
|
||||
@@ -50,12 +51,16 @@
|
||||
<string name="editor_save">Save</string>
|
||||
<string name="editor_saved_toast">Settings saved</string>
|
||||
<string name="editor_invalid_value_toast">Enter a valid number for every field</string>
|
||||
<string name="pose_phase_starting_stance">Starting Stance</string>
|
||||
<string name="pose_phase_approach">Approach</string>
|
||||
<string name="pose_phase_pushaway">Pushaway</string>
|
||||
<string name="pose_phase_back_swing">Backswing</string>
|
||||
<string name="pose_phase_starting_stance">Starting Position</string>
|
||||
<string name="pose_phase_approach">½ Step / Preparation Step</string>
|
||||
<string name="pose_phase_pushaway">Push Away & Backswing</string>
|
||||
<string name="pose_phase_back_swing">@string/pose_phase_pushaway</string>
|
||||
<string name="pose_phase_power_step">Power Step</string>
|
||||
<string name="pose_phase_slide_and_release">Slide & Release</string>
|
||||
<string name="pose_phase_waiting">Waiting for stance…</string>
|
||||
<string name="pose_phase_slide_and_release">Slide & Finishing Position</string>
|
||||
<string name="pose_phase_waiting">Waiting for starting position…</string>
|
||||
<string name="step_term_half_step">½ Step</string>
|
||||
<string name="step_term_preparation_step">Preparation Step</string>
|
||||
<string name="step_term_slide">Slide</string>
|
||||
<string name="step_term_finishing_position">Finishing Position</string>
|
||||
<string name="pose_metrics_format">Torso: %1$s · Knee L: %2$s R: %3$s\nElbow L: %4$s R: %5$s</string>
|
||||
</resources>
|
||||
@@ -0,0 +1,102 @@
|
||||
# Architecture
|
||||
|
||||
## Technology stack
|
||||
|
||||
| Layer | Choice |
|
||||
|---|---|
|
||||
| Platform | Android (min SDK 24, target/compile SDK 36) |
|
||||
| Languages | Kotlin (feature code), Java (legacy `MainActivity` entry point), C++ (native game shell) |
|
||||
| Build | Gradle 9.x (version catalog), CMake 3.22.1 via Android's `externalNativeBuild` |
|
||||
| Camera capture | CameraX (`core`, `camera2`, `lifecycle`, `video`, `view`, `effects`) |
|
||||
| Pose detection | Google ML Kit Pose Detection (accurate model) |
|
||||
| Concurrency | Kotlin Coroutines |
|
||||
| UI (bowling feature) | Android Views + ViewBinding, custom `View`s for the pose overlay |
|
||||
| UI (menu shell) | Native OpenGL ES 3.0, rendered from C++ via a `GLSurfaceView` |
|
||||
| Rendering (native) | Custom C++ `GLRenderer` / `UIRenderer` |
|
||||
|
||||
## High-level component map
|
||||
|
||||
The app is really two loosely-coupled subsystems living in one Gradle module (`app/`), connected by a single JNI call:
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Native["Native game shell (C++, JNI, OpenGL ES 3.0)"]
|
||||
GSM[GameStateManager]
|
||||
States[States: MainMenu / Menu1 / Menu2 / Menu3 / Settings]
|
||||
UIR[UIRenderer / GLRenderer]
|
||||
PB[PlatformBridge]
|
||||
GSM --> States
|
||||
GSM --> UIR
|
||||
States --> PB
|
||||
end
|
||||
|
||||
subgraph AndroidHost["Android host (Java/Kotlin)"]
|
||||
MA[MainActivity\nGLSurfaceView host]
|
||||
end
|
||||
|
||||
subgraph Bowling["Bowling capture + analysis (Kotlin)"]
|
||||
BCA[BowlingCameraActivity]
|
||||
CVM[CameraViewModel]
|
||||
CXC[CameraXController]
|
||||
PA[PoseAnalyzer]
|
||||
LSD[LiveStepDetector]
|
||||
PPD[PosePhaseDetector]
|
||||
PAC[PoseAngleCalculator]
|
||||
PLS[PoseLandmarkSmoother /\nAnkleHipMovingAverageFilter]
|
||||
POV[PoseOverlayView /\nPoseSkeletonRenderer]
|
||||
FUI[FeedbackUI / StepCounterUiController]
|
||||
DSL[DebugSessionLogger]
|
||||
PEA[ParameterEditorActivity /\nDetectorSettings]
|
||||
AA[AdminAuth /\nAdminLoginPrompt]
|
||||
end
|
||||
|
||||
MLKit[(ML Kit Pose Detection)]
|
||||
|
||||
PB -- "JNI: launchBowlingCamera()" --> MA
|
||||
MA -- "startActivity()" --> BCA
|
||||
BCA --> CVM --> CXC
|
||||
CXC -- camera frames --> PA
|
||||
PA -- landmarks --> MLKit
|
||||
MLKit -- pose result --> PA
|
||||
PA --> PLS --> LSD
|
||||
PA --> PAC
|
||||
LSD --> PPD
|
||||
PPD --> FUI
|
||||
PAC --> FUI
|
||||
PA --> POV
|
||||
CVM --> DSL
|
||||
PEA --> CVM
|
||||
AA --> PEA
|
||||
```
|
||||
|
||||
## Native game shell
|
||||
|
||||
- **`GameStateManager`** (singleton) owns the current `GameState` and drives `Update()` / `Render()` each frame. State transitions are deferred (`RequestStateChange`) so a state can safely trigger its own replacement mid-frame (e.g. from a button click handled during `Render()`).
|
||||
- **States** (`MainMenuState`, `Menu1State`, `Menu2State`, `Menu3State`, `SettingsState`) implement the actual menu screens; `Menu3State` is the entry point into the bowling feature.
|
||||
- **`PlatformBridge`** is the seam between shared state-machine code and platform-specific "launch a native feature" hooks, so state code doesn't need `#ifdef`s for Android vs. desktop. On Android, `LaunchBowlingCamera()` calls back into `MainActivity` over JNI (caching a `JavaVM` + global activity ref from `initGL()`); on the Windows/GLFW build it's a no-op log.
|
||||
- **`MainActivity`** (Java) hosts a `GLSurfaceView` (OpenGL ES 3.0, `RENDERMODE_CONTINUOUSLY`, `setPreserveEGLContextOnPause(true)` so launching `BowlingCameraActivity` on top doesn't tear down and have to re-init the GL context/UI). Touch events are forwarded to native code (`nativeOnTouch`) for in-engine hit-testing.
|
||||
- **Cross-platform note:** `Platform.h` already defines both `PLATFORM_ANDROID` and `PLATFORM_WINDOWS` (GLFW) branches, and `main.cpp` has a GLFW desktop loop. Only the Android CMake/Gradle build is currently wired up — there is no standalone desktop build target yet, but the state-machine/rendering code is written to be portable.
|
||||
|
||||
## Bowling capture & pose analysis pipeline
|
||||
|
||||
1. **`BowlingCameraActivity`** hosts the camera preview and UI chrome (switch camera, back, recording indicator); orientation follows the device sensor.
|
||||
2. **`CameraXController`** wraps CameraX use cases (preview, video, image analysis) and exposes camera frames.
|
||||
3. **`PoseAnalyzer`** runs each frame through ML Kit's accurate Pose Detection model and produces a `PoseFrame`.
|
||||
4. Landmarks are smoothed (**`PoseLandmarkSmoother`**, **`AnkleHipMovingAverageFilter`**) before being consumed by:
|
||||
- **`LiveStepDetector`** / **`StepDetector`** / **`StepCountingSession`** — step counting during the approach.
|
||||
- **`PosePhaseDetector`** — segments the approach into delivery phases.
|
||||
- **`PoseAngleCalculator`** — computes joint angles at points of interest.
|
||||
5. **`PoseOverlayView`** + **`PoseSkeletonRenderer`** draw the live skeleton over the camera preview.
|
||||
6. **`FeedbackUI`** / **`StepCounterUiController`** surface step count, phase, and feedback to the user.
|
||||
7. **`CameraViewModel`** coordinates the above and survives configuration changes; **`DebugSessionLogger`** records session data for offline tuning.
|
||||
8. **`DetectorSettings`** (via **`ParameterEditorActivity`**, gated by **`AdminAuth`**/**`AdminLoginPrompt`**) allows adjusting detection thresholds without a rebuild, for tuning during development/testing.
|
||||
|
||||
## Known architectural gap
|
||||
|
||||
`BowlingCameraActivity` is reachable two ways today: directly (e.g. via `adb`/launcher shortcut) and from the native menu's `Menu3State` via `PlatformBridge`. Per in-code comments, the direct-launch path exists because the feature isn't yet fully wired into the menu's visual flow — this should converge as the menu integration matures.
|
||||
|
||||
## Build & deployment
|
||||
|
||||
- Single Gradle module (`app`), AGP + CMake (`externalNativeBuild`) for the native library, targeting `arm64-v8a`, `armeabi-v7a`, `x86`, `x86_64`.
|
||||
- No CI/CD pipeline exists yet — see `docs/deliverables.md` for the CI/CD deliverable and milestone target.
|
||||
- No backend/server component exists; the app is fully on-device (no network calls in the current codebase).
|
||||
@@ -0,0 +1,29 @@
|
||||
# Technical Deliverables & Ownership
|
||||
|
||||
> **TODO (team):** Fill in real owner names and confirm/adjust milestone dates. The breakdown below is a first-pass mapping of the *existing* codebase into deliverables, derived from the current architecture (see `architecture.md`), so the team has a concrete starting point to assign in Jira rather than starting from a blank page. Contributor branches observed in git history: `Gabriel`, `Harine`, `Khalil`, `QiYing`, `YongWei's-Branch`, `au-au`, `jingwen` — use these as a hint for who's already been working in which area, not as a final assignment.
|
||||
|
||||
## How to use this doc
|
||||
|
||||
Each deliverable below should become one **Jira Epic**. Break each into Stories/Tasks under that epic, assign an owner per story, and link the epic back to the relevant row here (add the Jira link in the "Jira Epic" column once created).
|
||||
|
||||
## Deliverables
|
||||
|
||||
| # | Deliverable | Description | Owner | Jira Epic | M1 | M2 | M3 |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | **Native menu shell** | OpenGL ES 3.0 game-state menu (`GameStateManager`, states, `UIRenderer`, `PlatformBridge`) | TBD | TBD | Menu states render and navigate; touch input works | `Menu3State` fully launches bowling flow (menu integration gap closed) | Polish: transitions, settings persistence, no dropped frames |
|
||||
| 2 | **Camera capture pipeline** | CameraX integration (`CameraXController`, `CameraViewModel`, `CameraPermissions`) | TBD | TBD | Preview + front/back switch working on a real device | Stable recording with audio track; portrait + landscape | Robustness across device models; error handling for camera/permission edge cases |
|
||||
| 3 | **Pose detection & landmark processing** | ML Kit integration, smoothing (`PoseAnalyzer`, `PoseLandmarkSmoother`, `AnkleHipMovingAverageFilter`, `PoseFrame`) | TBD | TBD | Raw landmarks streamed from camera frames | Smoothing tuned to reduce jitter | Accuracy validated against recorded test sessions |
|
||||
| 4 | **Step & phase detection** | `LiveStepDetector`, `StepDetector`, `StepCountingSession`, `PosePhaseDetector` | TBD | TBD | Step count works for a straight-line approach | Phase segmentation (stance/approach/release) accurate for standard 4-5 step approaches | Edge cases (different approach lengths/styles) handled |
|
||||
| 5 | **Joint angle analysis** | `PoseAngleCalculator` | TBD | TBD | Ankle/hip angle computed at one key frame | Angles computed across full phase set | Feedback thresholds tuned against real coaching input |
|
||||
| 6 | **Pose overlay rendering** | `PoseOverlayView`, `PoseSkeletonRenderer` | TBD | TBD | Skeleton draws over live preview | Overlay stays in sync at speed/rotation | Visual polish (styling, confidence-based rendering) |
|
||||
| 7 | **Session feedback UI** | `FeedbackUI`, `StepCounterUiController` | TBD | TBD | Step count visible on screen | Phase + angle feedback surfaced live | Full session summary screen |
|
||||
| 8 | **Debug/session logging & tuning tools** | `DebugSessionLogger`, `DetectorSettings`, `ParameterEditorActivity`, `AdminAuth`/`AdminLoginPrompt` | TBD | TBD | Session data logged to file | Parameter editor allows live threshold tuning | Admin gating hardened; logs exportable for review |
|
||||
| 9 | **Documentation & repo hygiene** | `README.md`, `docs/`, `.gitignore`, repo structure | Harine (initial scaffold) | TBD | Docs skeleton + README exist (this commit) | Architecture/deliverables kept in sync with code changes | Docs reviewed each milestone; onboarding-tested by a teammate |
|
||||
| 10 | **CI/CD** | Automated build/test pipeline (`.github/workflows/`) | TBD | TBD | Basic workflow: build APK on push/PR | Add unit test run to pipeline | Add instrumented test run and/or lint/static analysis gate |
|
||||
| 11 | **Testing** | Unit + instrumented tests (`app/src/test`, `app/src/androidTest`) | TBD | TBD | Replace template stub tests with real coverage of at least one detector | Coverage for step/phase detection logic | Coverage for camera/UI integration paths where feasible |
|
||||
|
||||
## Milestone definitions (fill in actual dates)
|
||||
|
||||
- **M1 — Target date: TBD:** Core pipeline demonstrable end-to-end (capture → pose → step count) on a real device, even if rough.
|
||||
- **M2 — Target date: TBD:** Feature-complete for the core bowling analysis flow (phases, angles, feedback UI), menu integration closed, basic CI running.
|
||||
- **M3 — Target date: TBD:** Polish, testing, and hardening pass; documentation finalized; ready for demo/submission.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Product Design
|
||||
|
||||
## Overview
|
||||
|
||||
PinPoint is a mobile coaching aid for tenpin bowlers. It watches a bowler's approach through the phone's camera, tracks their body in real time using pose detection, and reports on the mechanics of their delivery — steps taken, timing, and joint angles at key phases — so a bowler (or their coach) can spot form issues without a human observer.
|
||||
|
||||
## Current features (as implemented)
|
||||
|
||||
- **Live camera capture** of the bowler's approach (front or back camera, switchable mid-session), landscape or portrait.
|
||||
- **Real-time pose overlay** — a skeleton drawn over the camera preview from detected body landmarks.
|
||||
- **Step detection** — counts steps taken during the approach from landmark motion.
|
||||
- **Delivery phase detection** — segments the approach into phases (e.g. stance, approach steps, release) for phase-specific analysis.
|
||||
- **Joint angle calculation** — computes relevant joint angles (e.g. ankle/hip) at points of interest.
|
||||
- **Landmark smoothing** — filters raw pose landmarks (moving-average / smoothing) to reduce jitter before analysis.
|
||||
- **Session feedback UI** — surfaces step count, phase, and feedback to the user during/after a session.
|
||||
- **Debug session logging** — records session data for later review/tuning of the detection logic.
|
||||
- **Adjustable detector parameters** — an admin-gated screen for tuning detection thresholds during development/testing.
|
||||
- **Native menu shell** — a separate OpenGL-based menu/game-state system (`MainActivity`), currently not yet wired to launch directly into the bowling camera flow from the UI (see `docs/architecture.md`).
|
||||
|
||||
## Target users
|
||||
|
||||
- Individual bowlers wanting self-guided form feedback without a coach present.
|
||||
- Coaches wanting a quick, repeatable way to capture and review a bowler's mechanics.
|
||||
|
||||
## Out of scope (current version)
|
||||
|
||||
- Cloud sync / multi-device history.
|
||||
- Automated scoring against a "correct form" reference model.
|
||||
- iOS support.
|
||||
|
||||
## UI/UX
|
||||
|
||||
See [`ui-ux/`](ui-ux/) for wireframes, screen flows, and mockups. As of this commit that folder is a placeholder — the team should add:
|
||||
|
||||
- A flow diagram of the menu → bowling capture screen navigation.
|
||||
- Screenshots or mockups of: main menu, bowling camera screen (portrait + landscape), settings/parameter editor, and feedback UI.
|
||||
@@ -0,0 +1,10 @@
|
||||
# UI/UX assets
|
||||
|
||||
Place wireframes, user-flow diagrams, and mockups for PinPoint here (images, exported PDFs, or links to a design tool like Figma).
|
||||
|
||||
This folder is currently empty — add assets covering at least:
|
||||
|
||||
- Menu navigation flow (`MainActivity` native menu states → `BowlingCameraActivity`)
|
||||
- Bowling camera screen, portrait and landscape layouts
|
||||
- Feedback/step-counter UI overlay
|
||||
- Settings / parameter editor screen
|
||||
@@ -0,0 +1,16 @@
|
||||
# Project Proposal
|
||||
|
||||
> **TODO:** Paste in the team's original submitted proposal here (verbatim, or a cleaned-up version with the same content). This file was scaffolded automatically as part of the required `docs/` structure and does not yet contain the actual submitted proposal text.
|
||||
|
||||
## Suggested sections to include
|
||||
|
||||
- Problem statement / motivation
|
||||
- Target users
|
||||
- Proposed solution and key features
|
||||
- Scope (in-scope vs. out-of-scope for this project)
|
||||
- Success criteria
|
||||
- Team members and roles
|
||||
|
||||
## Working summary (placeholder, derived from the current codebase — replace with the real proposal)
|
||||
|
||||
PinPoint (working name "BowlEye" in code) is an Android app that helps bowlers improve their approach and delivery form by recording video through the phone camera, running on-device pose detection (ML Kit) to track body landmarks in real time, and giving feedback on step timing, joint angles, and delivery phase.
|
||||
Reference in New Issue
Block a user