diff --git a/.gitignore b/.gitignore index dbbf9fe..0d5a5e0 100644 --- a/.gitignore +++ b/.gitignore @@ -1,20 +1,35 @@ -*.iml -.gradle -/local.properties -app/.idea/ -/.idea/caches -/.idea/libraries -/.idea/modules.xml -/.idea/workspace.xml -/.idea/navEditor.xml -/.idea/assetWizardSettings.xml -.DS_Store +# Build output /build /captures .externalNativeBuild .cxx -local.properties -/.idea -gradle\wrapper\gradle-wrapper.properties -gradle\libs.versions.toml \ No newline at end of file +# Gradle +.gradle + +# Local machine config (never commit — contains local SDK paths/secrets) +local.properties +/local.properties + +# Android Studio / IntelliJ +*.iml +/.idea +app/.idea/ +.idea/caches +.idea/libraries +.idea/modules.xml +.idea/workspace.xml +.idea/navEditor.xml +.idea/assetWizardSettings.xml +.idea/deploymentTargetSelector.xml +.idea/deviceManager.xml + +# Kotlin daemon cache +.kotlin + +# Visual Studio (native/CMake tooling) +.vs/ + +# OS +.DS_Store +Thumbs.db \ No newline at end of file diff --git a/.idea/deploymentTargetSelector.xml b/.idea/deploymentTargetSelector.xml deleted file mode 100644 index a8552ee..0000000 --- a/.idea/deploymentTargetSelector.xml +++ /dev/null @@ -1,22 +0,0 @@ - - - - - - - - - - - \ No newline at end of file diff --git a/.idea/deviceManager.xml b/.idea/deviceManager.xml deleted file mode 100644 index 91f9558..0000000 --- a/.idea/deviceManager.xml +++ /dev/null @@ -1,13 +0,0 @@ - - - - - - \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..e1c5969 --- /dev/null +++ b/README.md @@ -0,0 +1,84 @@ +# PinPoint (BowlEye) + +An Android application for analyzing a bowler's approach and delivery form using on-device pose detection. The app captures video of a bowler via the phone camera, tracks body landmarks in real time, and surfaces step-count, joint-angle, and phase feedback to help improve technique. + +See [`docs/`](docs/) for the project proposal, architecture, and deliverables/ownership breakdown. + +## Tech stack + +- **Language:** Kotlin (application/feature code), Java (legacy entry point), C++ (native game shell) +- **Platform:** Android (min SDK 24, target/compile SDK 36) +- **Build system:** Gradle (Kotlin DSL version catalog) + CMake 3.22.1 for the native module +- **Key libraries:** CameraX (capture), ML Kit Pose Detection — accurate model (landmark tracking), Kotlin Coroutines, AndroidX Lifecycle/ViewModel + +## Prerequisites + +- **Android Studio** (current stable channel) — bundles a compatible JDK, so no separate JDK install is required +- **Android SDK** with: + - Android SDK Platform 36 (and 36.1) + - NDK (side by side) — a recent version compatible with AGP + - CMake 3.22.1 + - Android SDK Build-Tools 36.0.0 +- A physical Android device with a camera (recommended) or an emulator with a virtual/webcam camera. Real camera input is strongly recommended since pose detection needs an actual moving subject. + +> The NDK/CMake/platform components above do **not** need to be installed manually — both Android Studio's "install missing components" prompt and a plain `./gradlew` command-line build will fetch them automatically on first sync/build, provided you have internet access and accept the SDK license prompts. + +## Getting started + +### Option A — Android Studio + +1. `File > Open` and select the repository root. +2. Let Gradle sync; approve any "install missing SDK components" prompts. +3. Connect a device (enable **Developer Options > USB debugging** on the phone) or start an emulator. +4. Click **Run** ▶ with the `app` configuration selected. + +### Option B — Command line + +```bash +# Windows +.\gradlew.bat assembleDebug +.\gradlew.bat installDebug + +# macOS / Linux +./gradlew assembleDebug +./gradlew installDebug +``` + +Then launch an activity directly, e.g.: + +```bash +adb shell am start -n com.example.jnicpp/.MainActivity +adb shell am start -n com.example.jnicpp/.bowling.BowlingCameraActivity +``` + +### First-time local setup notes + +- Gradle needs a `local.properties` file **at the repository root** (not inside `app/`) pointing at your local Android SDK, e.g.: + ``` + sdk.dir=/Users/you/Library/Android/sdk + ``` + Android Studio creates/updates this automatically on first sync. This file is machine-specific and is gitignored — never commit it. +- The app requests **Camera** and **Record Audio** permissions at runtime; grant both to use the bowling analysis screen. + +## Project structure + +``` +app/ +├── src/main/cpp/ # Native C++ game shell (OpenGL ES menu, JNI bridge), built via CMake +├── src/main/java/.../ # MainActivity (native GL menu host) +└── src/main/java/.../bowling/ # Bowling capture + pose analysis feature (Kotlin, CameraX + ML Kit) +docs/ # Proposal, architecture, design, and deliverables documentation +``` + +The native GL menu (`MainActivity`) and the bowling camera/pose-analysis screen (`BowlingCameraActivity`) are currently two independent entry points — see [`docs/architecture.md`](docs/architecture.md) for how they're intended to connect. + +## Running tests + +```bash +./gradlew test # JVM unit tests +./gradlew connectedCheck # Instrumented tests (requires a connected device/emulator) +``` + +## Cross-platform notes + +The native game-state code under `app/src/main/cpp/my_gl_app/` is written behind a `Platform.h` seam with both `PLATFORM_ANDROID` and `PLATFORM_WINDOWS` (GLFW) code paths, so the menu/state-machine logic itself is portable. Only the Android (Gradle/CMake) build is wired up today; there is no standalone desktop build target yet. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..b656fb8 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,102 @@ +# Architecture + +## Technology stack + +| Layer | Choice | +|---|---| +| Platform | Android (min SDK 24, target/compile SDK 36) | +| Languages | Kotlin (feature code), Java (legacy `MainActivity` entry point), C++ (native game shell) | +| Build | Gradle 9.x (version catalog), CMake 3.22.1 via Android's `externalNativeBuild` | +| Camera capture | CameraX (`core`, `camera2`, `lifecycle`, `video`, `view`, `effects`) | +| Pose detection | Google ML Kit Pose Detection (accurate model) | +| Concurrency | Kotlin Coroutines | +| UI (bowling feature) | Android Views + ViewBinding, custom `View`s for the pose overlay | +| UI (menu shell) | Native OpenGL ES 3.0, rendered from C++ via a `GLSurfaceView` | +| Rendering (native) | Custom C++ `GLRenderer` / `UIRenderer` | + +## High-level component map + +The app is really two loosely-coupled subsystems living in one Gradle module (`app/`), connected by a single JNI call: + +```mermaid +flowchart TB + subgraph Native["Native game shell (C++, JNI, OpenGL ES 3.0)"] + GSM[GameStateManager] + States[States: MainMenu / Menu1 / Menu2 / Menu3 / Settings] + UIR[UIRenderer / GLRenderer] + PB[PlatformBridge] + GSM --> States + GSM --> UIR + States --> PB + end + + subgraph AndroidHost["Android host (Java/Kotlin)"] + MA[MainActivity\nGLSurfaceView host] + end + + subgraph Bowling["Bowling capture + analysis (Kotlin)"] + BCA[BowlingCameraActivity] + CVM[CameraViewModel] + CXC[CameraXController] + PA[PoseAnalyzer] + LSD[LiveStepDetector] + PPD[PosePhaseDetector] + PAC[PoseAngleCalculator] + PLS[PoseLandmarkSmoother /\nAnkleHipMovingAverageFilter] + POV[PoseOverlayView /\nPoseSkeletonRenderer] + FUI[FeedbackUI / StepCounterUiController] + DSL[DebugSessionLogger] + PEA[ParameterEditorActivity /\nDetectorSettings] + AA[AdminAuth /\nAdminLoginPrompt] + end + + MLKit[(ML Kit Pose Detection)] + + PB -- "JNI: launchBowlingCamera()" --> MA + MA -- "startActivity()" --> BCA + BCA --> CVM --> CXC + CXC -- camera frames --> PA + PA -- landmarks --> MLKit + MLKit -- pose result --> PA + PA --> PLS --> LSD + PA --> PAC + LSD --> PPD + PPD --> FUI + PAC --> FUI + PA --> POV + CVM --> DSL + PEA --> CVM + AA --> PEA +``` + +## Native game shell + +- **`GameStateManager`** (singleton) owns the current `GameState` and drives `Update()` / `Render()` each frame. State transitions are deferred (`RequestStateChange`) so a state can safely trigger its own replacement mid-frame (e.g. from a button click handled during `Render()`). +- **States** (`MainMenuState`, `Menu1State`, `Menu2State`, `Menu3State`, `SettingsState`) implement the actual menu screens; `Menu3State` is the entry point into the bowling feature. +- **`PlatformBridge`** is the seam between shared state-machine code and platform-specific "launch a native feature" hooks, so state code doesn't need `#ifdef`s for Android vs. desktop. On Android, `LaunchBowlingCamera()` calls back into `MainActivity` over JNI (caching a `JavaVM` + global activity ref from `initGL()`); on the Windows/GLFW build it's a no-op log. +- **`MainActivity`** (Java) hosts a `GLSurfaceView` (OpenGL ES 3.0, `RENDERMODE_CONTINUOUSLY`, `setPreserveEGLContextOnPause(true)` so launching `BowlingCameraActivity` on top doesn't tear down and have to re-init the GL context/UI). Touch events are forwarded to native code (`nativeOnTouch`) for in-engine hit-testing. +- **Cross-platform note:** `Platform.h` already defines both `PLATFORM_ANDROID` and `PLATFORM_WINDOWS` (GLFW) branches, and `main.cpp` has a GLFW desktop loop. Only the Android CMake/Gradle build is currently wired up — there is no standalone desktop build target yet, but the state-machine/rendering code is written to be portable. + +## Bowling capture & pose analysis pipeline + +1. **`BowlingCameraActivity`** hosts the camera preview and UI chrome (switch camera, back, recording indicator); orientation follows the device sensor. +2. **`CameraXController`** wraps CameraX use cases (preview, video, image analysis) and exposes camera frames. +3. **`PoseAnalyzer`** runs each frame through ML Kit's accurate Pose Detection model and produces a `PoseFrame`. +4. Landmarks are smoothed (**`PoseLandmarkSmoother`**, **`AnkleHipMovingAverageFilter`**) before being consumed by: + - **`LiveStepDetector`** / **`StepDetector`** / **`StepCountingSession`** — step counting during the approach. + - **`PosePhaseDetector`** — segments the approach into delivery phases. + - **`PoseAngleCalculator`** — computes joint angles at points of interest. +5. **`PoseOverlayView`** + **`PoseSkeletonRenderer`** draw the live skeleton over the camera preview. +6. **`FeedbackUI`** / **`StepCounterUiController`** surface step count, phase, and feedback to the user. +7. **`CameraViewModel`** coordinates the above and survives configuration changes; **`DebugSessionLogger`** records session data for offline tuning. +8. **`DetectorSettings`** (via **`ParameterEditorActivity`**, gated by **`AdminAuth`**/**`AdminLoginPrompt`**) allows adjusting detection thresholds without a rebuild, for tuning during development/testing. + +## Known architectural gap + +`BowlingCameraActivity` is reachable two ways today: directly (e.g. via `adb`/launcher shortcut) and from the native menu's `Menu3State` via `PlatformBridge`. Per in-code comments, the direct-launch path exists because the feature isn't yet fully wired into the menu's visual flow — this should converge as the menu integration matures. + +## Build & deployment + +- Single Gradle module (`app`), AGP + CMake (`externalNativeBuild`) for the native library, targeting `arm64-v8a`, `armeabi-v7a`, `x86`, `x86_64`. +- No CI/CD pipeline exists yet — see `docs/deliverables.md` for the CI/CD deliverable and milestone target. +- No backend/server component exists; the app is fully on-device (no network calls in the current codebase). diff --git a/docs/deliverables.md b/docs/deliverables.md new file mode 100644 index 0000000..f1229b6 --- /dev/null +++ b/docs/deliverables.md @@ -0,0 +1,29 @@ +# Technical Deliverables & Ownership + +> **TODO (team):** Fill in real owner names and confirm/adjust milestone dates. The breakdown below is a first-pass mapping of the *existing* codebase into deliverables, derived from the current architecture (see `architecture.md`), so the team has a concrete starting point to assign in Jira rather than starting from a blank page. Contributor branches observed in git history: `Gabriel`, `Harine`, `Khalil`, `QiYing`, `YongWei's-Branch`, `au-au`, `jingwen` — use these as a hint for who's already been working in which area, not as a final assignment. + +## How to use this doc + +Each deliverable below should become one **Jira Epic**. Break each into Stories/Tasks under that epic, assign an owner per story, and link the epic back to the relevant row here (add the Jira link in the "Jira Epic" column once created). + +## Deliverables + +| # | Deliverable | Description | Owner | Jira Epic | M1 | M2 | M3 | +|---|---|---|---|---|---|---|---| +| 1 | **Native menu shell** | OpenGL ES 3.0 game-state menu (`GameStateManager`, states, `UIRenderer`, `PlatformBridge`) | TBD | TBD | Menu states render and navigate; touch input works | `Menu3State` fully launches bowling flow (menu integration gap closed) | Polish: transitions, settings persistence, no dropped frames | +| 2 | **Camera capture pipeline** | CameraX integration (`CameraXController`, `CameraViewModel`, `CameraPermissions`) | TBD | TBD | Preview + front/back switch working on a real device | Stable recording with audio track; portrait + landscape | Robustness across device models; error handling for camera/permission edge cases | +| 3 | **Pose detection & landmark processing** | ML Kit integration, smoothing (`PoseAnalyzer`, `PoseLandmarkSmoother`, `AnkleHipMovingAverageFilter`, `PoseFrame`) | TBD | TBD | Raw landmarks streamed from camera frames | Smoothing tuned to reduce jitter | Accuracy validated against recorded test sessions | +| 4 | **Step & phase detection** | `LiveStepDetector`, `StepDetector`, `StepCountingSession`, `PosePhaseDetector` | TBD | TBD | Step count works for a straight-line approach | Phase segmentation (stance/approach/release) accurate for standard 4-5 step approaches | Edge cases (different approach lengths/styles) handled | +| 5 | **Joint angle analysis** | `PoseAngleCalculator` | TBD | TBD | Ankle/hip angle computed at one key frame | Angles computed across full phase set | Feedback thresholds tuned against real coaching input | +| 6 | **Pose overlay rendering** | `PoseOverlayView`, `PoseSkeletonRenderer` | TBD | TBD | Skeleton draws over live preview | Overlay stays in sync at speed/rotation | Visual polish (styling, confidence-based rendering) | +| 7 | **Session feedback UI** | `FeedbackUI`, `StepCounterUiController` | TBD | TBD | Step count visible on screen | Phase + angle feedback surfaced live | Full session summary screen | +| 8 | **Debug/session logging & tuning tools** | `DebugSessionLogger`, `DetectorSettings`, `ParameterEditorActivity`, `AdminAuth`/`AdminLoginPrompt` | TBD | TBD | Session data logged to file | Parameter editor allows live threshold tuning | Admin gating hardened; logs exportable for review | +| 9 | **Documentation & repo hygiene** | `README.md`, `docs/`, `.gitignore`, repo structure | Harine (initial scaffold) | TBD | Docs skeleton + README exist (this commit) | Architecture/deliverables kept in sync with code changes | Docs reviewed each milestone; onboarding-tested by a teammate | +| 10 | **CI/CD** | Automated build/test pipeline (`.github/workflows/`) | TBD | TBD | Basic workflow: build APK on push/PR | Add unit test run to pipeline | Add instrumented test run and/or lint/static analysis gate | +| 11 | **Testing** | Unit + instrumented tests (`app/src/test`, `app/src/androidTest`) | TBD | TBD | Replace template stub tests with real coverage of at least one detector | Coverage for step/phase detection logic | Coverage for camera/UI integration paths where feasible | + +## Milestone definitions (fill in actual dates) + +- **M1 — Target date: TBD:** Core pipeline demonstrable end-to-end (capture → pose → step count) on a real device, even if rough. +- **M2 — Target date: TBD:** Feature-complete for the core bowling analysis flow (phases, angles, feedback UI), menu integration closed, basic CI running. +- **M3 — Target date: TBD:** Polish, testing, and hardening pass; documentation finalized; ready for demo/submission. diff --git a/docs/design/product-design.md b/docs/design/product-design.md new file mode 100644 index 0000000..bf3f8a8 --- /dev/null +++ b/docs/design/product-design.md @@ -0,0 +1,36 @@ +# Product Design + +## Overview + +PinPoint is a mobile coaching aid for tenpin bowlers. It watches a bowler's approach through the phone's camera, tracks their body in real time using pose detection, and reports on the mechanics of their delivery — steps taken, timing, and joint angles at key phases — so a bowler (or their coach) can spot form issues without a human observer. + +## Current features (as implemented) + +- **Live camera capture** of the bowler's approach (front or back camera, switchable mid-session), landscape or portrait. +- **Real-time pose overlay** — a skeleton drawn over the camera preview from detected body landmarks. +- **Step detection** — counts steps taken during the approach from landmark motion. +- **Delivery phase detection** — segments the approach into phases (e.g. stance, approach steps, release) for phase-specific analysis. +- **Joint angle calculation** — computes relevant joint angles (e.g. ankle/hip) at points of interest. +- **Landmark smoothing** — filters raw pose landmarks (moving-average / smoothing) to reduce jitter before analysis. +- **Session feedback UI** — surfaces step count, phase, and feedback to the user during/after a session. +- **Debug session logging** — records session data for later review/tuning of the detection logic. +- **Adjustable detector parameters** — an admin-gated screen for tuning detection thresholds during development/testing. +- **Native menu shell** — a separate OpenGL-based menu/game-state system (`MainActivity`), currently not yet wired to launch directly into the bowling camera flow from the UI (see `docs/architecture.md`). + +## Target users + +- Individual bowlers wanting self-guided form feedback without a coach present. +- Coaches wanting a quick, repeatable way to capture and review a bowler's mechanics. + +## Out of scope (current version) + +- Cloud sync / multi-device history. +- Automated scoring against a "correct form" reference model. +- iOS support. + +## UI/UX + +See [`ui-ux/`](ui-ux/) for wireframes, screen flows, and mockups. As of this commit that folder is a placeholder — the team should add: + +- A flow diagram of the menu → bowling capture screen navigation. +- Screenshots or mockups of: main menu, bowling camera screen (portrait + landscape), settings/parameter editor, and feedback UI. diff --git a/docs/design/ui-ux/README.md b/docs/design/ui-ux/README.md new file mode 100644 index 0000000..26dd178 --- /dev/null +++ b/docs/design/ui-ux/README.md @@ -0,0 +1,10 @@ +# UI/UX assets + +Place wireframes, user-flow diagrams, and mockups for PinPoint here (images, exported PDFs, or links to a design tool like Figma). + +This folder is currently empty — add assets covering at least: + +- Menu navigation flow (`MainActivity` native menu states → `BowlingCameraActivity`) +- Bowling camera screen, portrait and landscape layouts +- Feedback/step-counter UI overlay +- Settings / parameter editor screen diff --git a/docs/proposal.md b/docs/proposal.md new file mode 100644 index 0000000..6947543 --- /dev/null +++ b/docs/proposal.md @@ -0,0 +1,16 @@ +# Project Proposal + +> **TODO:** Paste in the team's original submitted proposal here (verbatim, or a cleaned-up version with the same content). This file was scaffolded automatically as part of the required `docs/` structure and does not yet contain the actual submitted proposal text. + +## Suggested sections to include + +- Problem statement / motivation +- Target users +- Proposed solution and key features +- Scope (in-scope vs. out-of-scope for this project) +- Success criteria +- Team members and roles + +## Working summary (placeholder, derived from the current codebase — replace with the real proposal) + +PinPoint (working name "BowlEye" in code) is an Android app that helps bowlers improve their approach and delivery form by recording video through the phone camera, running on-device pose detection (ML Kit) to track body landmarks in real time, and giving feedback on step timing, joint angles, and delivery phase.