unplayable01/PinPoint
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Squashed commit of the following:

Browse Source 
commit f0b9b2af67
Merge: 373d65a bbaa7bb
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Sat Sep 12 18:17:04 2026 +0800

    Merge branch 'merge-testing'

commit 373d65a9d5
Merge: 28fc652 dd0e6fd
Author: harine <harinesumen@gmail.com>
Date:   Sat Sep 12 17:34:27 2026 +0800

    Merge branch 'master' into Harine

commit 28fc652097
Author: harine <harinesumen@gmail.com>
Date:   Sat Sep 12 17:34:07 2026 +0800

    new stuff added yes

commit bbaa7bb04f
Merge: 1c6ced2 610bfd7
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 11:18:35 2026 +0800

    Merge branch 'Gabriel' into merge-testing

commit 1c6ced2dce
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 11:12:30 2026 +0800

    this good

commit dd0e6fdc39
Author: DefiantWanderer <yongwei1349@gmail.com>
Date:   Fri Sep 11 10:32:34 2026 +0800

    Revert "fixing compatibility issues"

    This reverts commit 564e7d83f8.

commit 564e7d83f8
Author: DefiantWanderer <yongwei1349@gmail.com>
Date:   Fri Sep 11 10:28:40 2026 +0800

    fixing compatibility issues

commit 610bfd7991
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 10:13:25 2026 +0800

    Merged Khalil

commit cfe76da1d3
Merge: 584818b 9fcdc13
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Fri Sep 11 10:03:51 2026 +0800

    Merge branch 'Khalil' into Gabriel

commit 584818b95a
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 12:47:06 2026 +0800

    added back phase toggle button

commit 17fad430c1
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 12:09:46 2026 +0800

    Cleaned up merge UI, added reset step button

commit ac4366d04d
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 00:31:47 2026 +0800

    Added some UI to recording

commit d601efcf43
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Thu Sep 10 00:13:06 2026 +0800

    Corrected 5 step approach phases & added placeholders for future logic

commit 7cf217488a
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Wed Sep 9 22:48:06 2026 +0800

    Cleared warnings

commit 981071ef74
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Wed Sep 9 20:09:18 2026 +0800

    Fix Merge, cleaned up UI

commit f30bbe2a34
Merge: 2f45115 7518594
Author: Gabriel Low <73009315+gabriellow1111@users.noreply.github.com>
Date:   Tue Sep 8 14:11:21 2026 +0800

    Merge branch 'au-au' into Gabriel

commit 9fcdc13699
Author: Khalil Belabadia <belabakhalil@gmail.com>
Date:   Mon Sep 7 22:16:00 2026 +0800

    Audio feedback update

    Added the following files :

    - AudioCue.kt
    - AudioFeedbackEngine
    - AudioFeedbackSettings

    Edited files :

    - BowlingCameraActivity (added the triggerAudiofeedback function)

commit 7518594440
Merge: e7a9c2b 7af128d
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 21:26:18 2026 +0800

    Merge remote-tracking branch 'origin/au-au' into au-au

    # Conflicts:
    #	app/src/main/java/com/example/jnicpp/bowling/BowlingCameraActivity.kt
    #	app/src/main/res/layout-land/activity_bowling_camera.xml
    #	app/src/main/res/layout/activity_bowling_camera.xml

commit e7a9c2b140
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 21:22:11 2026 +0800

    Add live per-step form feedback based on joint angles

    Extends pose angle tracking with knee bend (hip-knee-ankle), then uses
    it alongside the existing elbow/shoulder angles in a new
    PoseStageAdvisor to give a short live cue for whichever step of the
    approach is in progress: push-away on step 2, downswing on step 3,
    backswing on step 4, and knee-bend/arm-extension on the final step.
    Thresholds are starting defaults, not measured coaching data, and are
    expected to be retuned against real approach footage.

    Also fixes a layout bug found while testing this live: the new
    feedback text was chained via ConstraintLayout's toBottomOf to the
    step banner above it, so whenever that banner was hidden (GONE) the
    feedback text rendered at its collapsed zero-height position instead
    of staying put, landing on top of the recording indicator. Both now
    sit in a plain vertical LinearLayout, which collapses GONE children
    correctly.

    Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

commit 7af128d575
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 20:58:03 2026 +0800

    steps

    identify the steps

commit 2bb7c6ca7b
Author: midnight-masala <2401021@sit.singaporetech.edu.sg>
Date:   Mon Sep 7 20:58:03 2026 +0800

    Add live final-position feedback and fix step detection accuracy

    Shows a live banner as each step of the approach is counted, ending in
    "FINAL POSITION - RELEASE!" once the 5th footfall lands. Also fixes
    LiveStepDetector's peak-prominence check, which compared a candidate
    footfall only to its immediate neighboring frame and silently dropped
    real steps that landed across several closely-spaced frames; it now
    tracks the true rise/fall trough on each side instead. Switches
    PoseAnalyzer to ML Kit's faster base pose model (the accurate model was
    starving the peak detector of frames on unaccelerated hardware), and
    makes LiveStepDetector's stillness-based reset toggleable, currently
    off for easier testing against recorded reference clips.

    Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
jingwen121
2026-09-12 21:43:10 +08:00
parent b907747755
commit c6501d00f2
44 changed files with 1757 additions and 1063 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+102
View File
@@ -0,0 +1,102 @@
# Architecture
## Technology stack
| Layer | Choice |
|---|---|
| Platform | Android (min SDK 24, target/compile SDK 36) |
| Languages | Kotlin (feature code), Java (legacy `MainActivity` entry point), C++ (native game shell) |
| Build | Gradle 9.x (version catalog), CMake 3.22.1 via Android's `externalNativeBuild` |
| Camera capture | CameraX (`core`, `camera2`, `lifecycle`, `video`, `view`, `effects`) |
| Pose detection | Google ML Kit Pose Detection (accurate model) |
| Concurrency | Kotlin Coroutines |
| UI (bowling feature) | Android Views + ViewBinding, custom `View`s for the pose overlay |
| UI (menu shell) | Native OpenGL ES 3.0, rendered from C++ via a `GLSurfaceView` |
| Rendering (native) | Custom C++ `GLRenderer` / `UIRenderer` |
## High-level component map
The app is really two loosely-coupled subsystems living in one Gradle module (`app/`), connected by a single JNI call:
```mermaid
flowchart TB
subgraph Native["Native game shell (C++, JNI, OpenGL ES 3.0)"]
GSM[GameStateManager]
States[States: MainMenu / Menu1 / Menu2 / Menu3 / Settings]
UIR[UIRenderer / GLRenderer]
PB[PlatformBridge]
GSM --> States
GSM --> UIR
States --> PB
end
subgraph AndroidHost["Android host (Java/Kotlin)"]
MA[MainActivity\nGLSurfaceView host]
end
subgraph Bowling["Bowling capture + analysis (Kotlin)"]
BCA[BowlingCameraActivity]
CVM[CameraViewModel]
CXC[CameraXController]
PA[PoseAnalyzer]
LSD[LiveStepDetector]
PPD[PosePhaseDetector]
PAC[PoseAngleCalculator]
PLS[PoseLandmarkSmoother /\nAnkleHipMovingAverageFilter]
POV[PoseOverlayView /\nPoseSkeletonRenderer]
FUI[FeedbackUI / StepCounterUiController]
DSL[DebugSessionLogger]
PEA[ParameterEditorActivity /\nDetectorSettings]
AA[AdminAuth /\nAdminLoginPrompt]
end
MLKit[(ML Kit Pose Detection)]
PB -- "JNI: launchBowlingCamera()" --> MA
MA -- "startActivity()" --> BCA
BCA --> CVM --> CXC
CXC -- camera frames --> PA
PA -- landmarks --> MLKit
MLKit -- pose result --> PA
PA --> PLS --> LSD
PA --> PAC
LSD --> PPD
PPD --> FUI
PAC --> FUI
PA --> POV
CVM --> DSL
PEA --> CVM
AA --> PEA
```
## Native game shell
- **`GameStateManager`** (singleton) owns the current `GameState` and drives `Update()` / `Render()` each frame. State transitions are deferred (`RequestStateChange`) so a state can safely trigger its own replacement mid-frame (e.g. from a button click handled during `Render()`).
- **States** (`MainMenuState`, `Menu1State`, `Menu2State`, `Menu3State`, `SettingsState`) implement the actual menu screens; `Menu3State` is the entry point into the bowling feature.
- **`PlatformBridge`** is the seam between shared state-machine code and platform-specific "launch a native feature" hooks, so state code doesn't need `#ifdef`s for Android vs. desktop. On Android, `LaunchBowlingCamera()` calls back into `MainActivity` over JNI (caching a `JavaVM` + global activity ref from `initGL()`); on the Windows/GLFW build it's a no-op log.
- **`MainActivity`** (Java) hosts a `GLSurfaceView` (OpenGL ES 3.0, `RENDERMODE_CONTINUOUSLY`, `setPreserveEGLContextOnPause(true)` so launching `BowlingCameraActivity` on top doesn't tear down and have to re-init the GL context/UI). Touch events are forwarded to native code (`nativeOnTouch`) for in-engine hit-testing.
- **Cross-platform note:** `Platform.h` already defines both `PLATFORM_ANDROID` and `PLATFORM_WINDOWS` (GLFW) branches, and `main.cpp` has a GLFW desktop loop. Only the Android CMake/Gradle build is currently wired up — there is no standalone desktop build target yet, but the state-machine/rendering code is written to be portable.
## Bowling capture & pose analysis pipeline
1. **`BowlingCameraActivity`** hosts the camera preview and UI chrome (switch camera, back, recording indicator); orientation follows the device sensor.
2. **`CameraXController`** wraps CameraX use cases (preview, video, image analysis) and exposes camera frames.
3. **`PoseAnalyzer`** runs each frame through ML Kit's accurate Pose Detection model and produces a `PoseFrame`.
4. Landmarks are smoothed (**`PoseLandmarkSmoother`**, **`AnkleHipMovingAverageFilter`**) before being consumed by:
- **`LiveStepDetector`** / **`StepDetector`** / **`StepCountingSession`** — step counting during the approach.
- **`PosePhaseDetector`** — segments the approach into delivery phases.
- **`PoseAngleCalculator`** — computes joint angles at points of interest.
5. **`PoseOverlayView`** + **`PoseSkeletonRenderer`** draw the live skeleton over the camera preview.
6. **`FeedbackUI`** / **`StepCounterUiController`** surface step count, phase, and feedback to the user.
7. **`CameraViewModel`** coordinates the above and survives configuration changes; **`DebugSessionLogger`** records session data for offline tuning.
8. **`DetectorSettings`** (via **`ParameterEditorActivity`**, gated by **`AdminAuth`**/**`AdminLoginPrompt`**) allows adjusting detection thresholds without a rebuild, for tuning during development/testing.
## Known architectural gap
`BowlingCameraActivity` is reachable two ways today: directly (e.g. via `adb`/launcher shortcut) and from the native menu's `Menu3State` via `PlatformBridge`. Per in-code comments, the direct-launch path exists because the feature isn't yet fully wired into the menu's visual flow — this should converge as the menu integration matures.
## Build & deployment
- Single Gradle module (`app`), AGP + CMake (`externalNativeBuild`) for the native library, targeting `arm64-v8a`, `armeabi-v7a`, `x86`, `x86_64`.
- No CI/CD pipeline exists yet — see `docs/deliverables.md` for the CI/CD deliverable and milestone target.
- No backend/server component exists; the app is fully on-device (no network calls in the current codebase).
docs/deliverables.md
+29
View File
@@ -0,0 +1,29 @@
# Technical Deliverables & Ownership
> **TODO (team):** Fill in real owner names and confirm/adjust milestone dates. The breakdown below is a first-pass mapping of the *existing* codebase into deliverables, derived from the current architecture (see `architecture.md`), so the team has a concrete starting point to assign in Jira rather than starting from a blank page. Contributor branches observed in git history: `Gabriel`, `Harine`, `Khalil`, `QiYing`, `YongWei's-Branch`, `au-au`, `jingwen` — use these as a hint for who's already been working in which area, not as a final assignment.
## How to use this doc
Each deliverable below should become one **Jira Epic**. Break each into Stories/Tasks under that epic, assign an owner per story, and link the epic back to the relevant row here (add the Jira link in the "Jira Epic" column once created).
## Deliverables
| # | Deliverable | Description | Owner | Jira Epic | M1 | M2 | M3 |
|---|---|---|---|---|---|---|---|
| 1 | **Native menu shell** | OpenGL ES 3.0 game-state menu (`GameStateManager`, states, `UIRenderer`, `PlatformBridge`) | TBD | TBD | Menu states render and navigate; touch input works | `Menu3State` fully launches bowling flow (menu integration gap closed) | Polish: transitions, settings persistence, no dropped frames |
| 2 | **Camera capture pipeline** | CameraX integration (`CameraXController`, `CameraViewModel`, `CameraPermissions`) | TBD | TBD | Preview + front/back switch working on a real device | Stable recording with audio track; portrait + landscape | Robustness across device models; error handling for camera/permission edge cases |
| 3 | **Pose detection & landmark processing** | ML Kit integration, smoothing (`PoseAnalyzer`, `PoseLandmarkSmoother`, `AnkleHipMovingAverageFilter`, `PoseFrame`) | TBD | TBD | Raw landmarks streamed from camera frames | Smoothing tuned to reduce jitter | Accuracy validated against recorded test sessions |
| 4 | **Step & phase detection** | `LiveStepDetector`, `StepDetector`, `StepCountingSession`, `PosePhaseDetector` | TBD | TBD | Step count works for a straight-line approach | Phase segmentation (stance/approach/release) accurate for standard 4-5 step approaches | Edge cases (different approach lengths/styles) handled |
| 5 | **Joint angle analysis** | `PoseAngleCalculator` | TBD | TBD | Ankle/hip angle computed at one key frame | Angles computed across full phase set | Feedback thresholds tuned against real coaching input |
| 6 | **Pose overlay rendering** | `PoseOverlayView`, `PoseSkeletonRenderer` | TBD | TBD | Skeleton draws over live preview | Overlay stays in sync at speed/rotation | Visual polish (styling, confidence-based rendering) |
| 7 | **Session feedback UI** | `FeedbackUI`, `StepCounterUiController` | TBD | TBD | Step count visible on screen | Phase + angle feedback surfaced live | Full session summary screen |
| 8 | **Debug/session logging & tuning tools** | `DebugSessionLogger`, `DetectorSettings`, `ParameterEditorActivity`, `AdminAuth`/`AdminLoginPrompt` | TBD | TBD | Session data logged to file | Parameter editor allows live threshold tuning | Admin gating hardened; logs exportable for review |
| 9 | **Documentation & repo hygiene** | `README.md`, `docs/`, `.gitignore`, repo structure | Harine (initial scaffold) | TBD | Docs skeleton + README exist (this commit) | Architecture/deliverables kept in sync with code changes | Docs reviewed each milestone; onboarding-tested by a teammate |
| 10 | **CI/CD** | Automated build/test pipeline (`.github/workflows/`) | TBD | TBD | Basic workflow: build APK on push/PR | Add unit test run to pipeline | Add instrumented test run and/or lint/static analysis gate |
| 11 | **Testing** | Unit + instrumented tests (`app/src/test`, `app/src/androidTest`) | TBD | TBD | Replace template stub tests with real coverage of at least one detector | Coverage for step/phase detection logic | Coverage for camera/UI integration paths where feasible |
## Milestone definitions (fill in actual dates)
- **M1 — Target date: TBD:** Core pipeline demonstrable end-to-end (capture → pose → step count) on a real device, even if rough.
- **M2 — Target date: TBD:** Feature-complete for the core bowling analysis flow (phases, angles, feedback UI), menu integration closed, basic CI running.
- **M3 — Target date: TBD:** Polish, testing, and hardening pass; documentation finalized; ready for demo/submission.
docs/design/product-design.md
+36
View File
@@ -0,0 +1,36 @@
# Product Design
## Overview
PinPoint is a mobile coaching aid for tenpin bowlers. It watches a bowler's approach through the phone's camera, tracks their body in real time using pose detection, and reports on the mechanics of their delivery — steps taken, timing, and joint angles at key phases — so a bowler (or their coach) can spot form issues without a human observer.
## Current features (as implemented)
- **Live camera capture** of the bowler's approach (front or back camera, switchable mid-session), landscape or portrait.
- **Real-time pose overlay** — a skeleton drawn over the camera preview from detected body landmarks.
- **Step detection** — counts steps taken during the approach from landmark motion.
- **Delivery phase detection** — segments the approach into phases (e.g. stance, approach steps, release) for phase-specific analysis.
- **Joint angle calculation** — computes relevant joint angles (e.g. ankle/hip) at points of interest.
- **Landmark smoothing** — filters raw pose landmarks (moving-average / smoothing) to reduce jitter before analysis.
- **Session feedback UI** — surfaces step count, phase, and feedback to the user during/after a session.
- **Debug session logging** — records session data for later review/tuning of the detection logic.
- **Adjustable detector parameters** — an admin-gated screen for tuning detection thresholds during development/testing.
- **Native menu shell** — a separate OpenGL-based menu/game-state system (`MainActivity`), currently not yet wired to launch directly into the bowling camera flow from the UI (see `docs/architecture.md`).
## Target users
- Individual bowlers wanting self-guided form feedback without a coach present.
- Coaches wanting a quick, repeatable way to capture and review a bowler's mechanics.
## Out of scope (current version)
- Cloud sync / multi-device history.
- Automated scoring against a "correct form" reference model.
- iOS support.
## UI/UX
See [`ui-ux/`](ui-ux/) for wireframes, screen flows, and mockups. As of this commit that folder is a placeholder — the team should add:
- A flow diagram of the menu → bowling capture screen navigation.
- Screenshots or mockups of: main menu, bowling camera screen (portrait + landscape), settings/parameter editor, and feedback UI.
docs/design/ui-ux/README.md
+10
View File
@@ -0,0 +1,10 @@
# UI/UX assets
Place wireframes, user-flow diagrams, and mockups for PinPoint here (images, exported PDFs, or links to a design tool like Figma).
This folder is currently empty — add assets covering at least:
- Menu navigation flow (`MainActivity` native menu states → `BowlingCameraActivity`)
- Bowling camera screen, portrait and landscape layouts
- Feedback/step-counter UI overlay
- Settings / parameter editor screen
docs/proposal.md
+16
View File
@@ -0,0 +1,16 @@
# Project Proposal
> **TODO:** Paste in the team's original submitted proposal here (verbatim, or a cleaned-up version with the same content). This file was scaffolded automatically as part of the required `docs/` structure and does not yet contain the actual submitted proposal text.
## Suggested sections to include
- Problem statement / motivation
- Target users
- Proposed solution and key features
- Scope (in-scope vs. out-of-scope for this project)
- Success criteria
- Team members and roles
## Working summary (placeholder, derived from the current codebase — replace with the real proposal)
PinPoint (working name "BowlEye" in code) is an Android app that helps bowlers improve their approach and delivery form by recording video through the phone camera, running on-device pose detection (ML Kit) to track body landmarks in real time, and giving feedback on step timing, joint angles, and delivery phase.