Merge branch 'master' of https://agile.bu.edu/gitlab/ec504/ec504_projects/group8 into detection

/dataSources/

/dataSources.local.xml

../.DS_store

../target/

 No newline at end of file

../src/target/

../.vscode/

 No newline at end of file

# Group8: Optimized Monocular VSLAM

# Documentation

**Nafis Abeer**: 

**Rohan Kumar**: roku@bu.edu

**Zane Mroue**:

**Samuel Gulinello**:

**Sanford Edelist**:

### Description

Visual Simultaneous Localization and Mapping (VSLAM) is the process of taking camera feed, as well as its position, and building a map of the current local world, specifically using visual input. This project uses this process, and builds upon it by also tracking objects within a frame. In this comes two problems: object detection, and then subsequent mapping and tracking of objects within a 3D space.

### Implementation

For simplicity, the general system framework is shown below:

```mermaid

graph LR;

    Z[Camera/Video] -->|Input| A

    A[VSLAM]-->|KeyFrames| B[YOLOv4];

    A-->|Features| C[Object Tracking];

    A-->|Features| E

    B-->|Object Detection|C;

    C-->|Objects| D[Database];

    D-->|Objects| E[GUI];

```

- **Camera/Video**: as of right now, we use prerecorded video as our examples and tests, but this system can be easily extended to real time camera systems or drone footage

  - the output are the frames that constitute the video

- **VSLAM**: using the MATLAB VSLAM algorithm, this process takes the raw frame and does two things: finds "features", or important points used for tracking, and finds "keyframes", which are a subset of the entire set of frames that capture the most information about the video (i.e. movement gives points their 3D position)

  - the outputs are the keyframes and the features

- **YOLOv4**: using a Java library, we perform deep learning using the YOLOv4 model, which is a convolutional neural network that takes a keyframe, and finds bounding boxes around each object that the model can discern

  - the output are the bounding boxes around each object for each frame

- **Object Tracking**: the significant contribution using data structures and algorithms, this system takes the bounding boxes of each object (in 2D on a single frame), and the features (in 3D on the same frame), finds each feature in each bounding box, and then tries to rectify the objects in the current frame with objects found in past frames. We solve this by implementing a data structure called an [ObjectSet](./src/main/java/object_detection/ObjectSet.java). For each object that has already been found, we compare a new object, and if there is some percentage of similarity in features contained in both objects, we combine these two objects and update the database correspondingly.

  - there is further explanation and runtime analysis in the Appendix A

  - the output is an iteratively more accurate set of objects that correspond to reality

- **Database**: for ease of retrieving, updating, and storing Objects and corresponding features, we use a MongoDB database

  - the output is storage for those objects from object tracking

- **GUI**: for an outward facing display of our work, we implemented a Javascript UI, that creates a server, such that we can view the system's output in any browser. 

  - the output is a clean point cloud view of objects and features that the camera has seen

### Features

# Code

The following links:

- [The branch and directory containing Java 17 Code to be executed](./../../tree/master/src/main/java)

- [The data needed for the examples used for this system](./../../tree/master/src/main/java/vslam/KeyFrames)

- [The testing code for this system](./../../tree/master/src/test/java/)

# Work Breakdown

**Nafis Abeer**:

**Rohan Kumar**:

**Zane Mroue**:

**Samuel Gulinello**:

**Sanford Edelist**:

# Appendix

### A: Runtime and Space Analysis of ObjectSet

> TODO

### B: References and Material Used

> need to fill in references and also ALL LIBRARIES USED (MATLAB, YOLO, Javascript stuff, etc)

## Working on this project:

- when creating a new branch, go into the Issues tab, click on the Issue you are working on, and create a branch off of that issue

- only change things in that branch, and then when finished we can all push into master

- within the group8 directory, create a subdirectory like:

  - group8/gui

  - group8/database

  - etc

- do your work within these subdirectories so we don't have issues resolving changes at the end

    <artifactId>vslam-objects</artifactId>

    <version>1.0</version>

    <properties>

    <!-- <properties>

        <maven.compiler.source>17</maven.compiler.source>

        <maven.compiler.target>17</maven.compiler.target>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

        <exec.mainClass>object_detection.ObjectDetector</exec.mainClass>

    </properties> -->

    <properties>

        <maven.compiler.source>17</maven.compiler.source>

        <maven.compiler.target>17</maven.compiler.target>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

        <exec.mainClass>yolo.YOLODetector</exec.mainClass>

    </properties>

    <dependencies>

        <!-- Spark framework for webdev -->

        <dependency>

            <groupId>com.sparkjava</groupId>

            <artifactId>spark-core</artifactId>

            <artifactId>log4j-slf4j-impl</artifactId>

            <version>2.20.0</version>

        </dependency>

        <!-- For JUnit (testing), with JUnit Jupiter -->

        <dependency>

            <groupId>org.junit.jupiter</groupId>

            <artifactId>junit-jupiter-api</artifactId>

            <version>5.7.0</version>

            <scope>test</scope>

        </dependency>

        <!-- MongoDB -->

        <dependency>

            <groupId>org.mongodb</groupId>

            <artifactId>mongodb-driver-sync</artifactId>

            <version>5.0.1</version>

        </dependency>

        <!-- JavaCV dependencies -->

        <dependency>

            <groupId>org.bytedeco</groupId>

            <artifactId>javacv</artifactId>

            <version>1.5.7</version>

        </dependency>

        <dependency>

            <groupId>org.bytedeco</groupId>

            <artifactId>javacv-platform</artifactId>

            <version>1.5.7</version>

        </dependency>

        <dependency>

            <groupId>org.bytedeco</groupId>

            <artifactId>opencv-platform</artifactId>

            <version>4.5.5-1.5.7</version>

        </dependency>

        <dependency>

            <groupId>org.bytedeco</groupId>

            <artifactId>javacpp</artifactId>

            <version>1.5.7</version>

        </dependency>

    </dependencies>

</project>

# Group8: Optimized Monocular VSLAM

## Using Maven

Maven is a tool to build and run Java projects. For this project we are using it to automate the process of installing dependencies and compiling the system.

The file named pom.xml in the main directory controls how maven builds the project on each run, and also holds dependencies. The most important fields are `properties`, which holds the information about JDK and execution, and `dependencies`, which holds information on external sources needed to compile the project. 

On each run, Maven will run test cases and compile into the target directory of the project. 

### Installation

For Mac you can just use `brew install maven`.

### Usage 

In the pom.xml file, under properties, we can add a line like the following:

`<exec.mainClass>object_detection.ObjectDetector</exec.mainClass>`

What this says is that when we call `mvn exec:java`, we search under src/java and execute the `main` method of the `ObjectDetector.java` class file. In this way we can test out any main method that we want.

The best way to use this in Intellij is by creating a build configuration using the following steps:

1. In the top right, find the dropdown of the `Run` options. (this should be the left of the play button and the debug button)

2. Press `Edit Configurations...`

3. Press `Add new configuration`

4. Find the maven icon and create this build config.

5. Give the config a name, and then under `Run`, add `clean compile exec:java`, which does each command in succession.

### Resources

- https://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

- https://maven.apache.org/run-maven/index.html 

## Working on this project:

- when creating a new branch, go into the Issues tab, click on the Issue you are working on, and create a branch off of that issue

- only change things in that branch, and then when finished we can all push into master

- within the group8 directory, create a subdirectory like:

  - group8/gui

  - group8/database

  - etc

- do your work within these subdirectories so we don't have issues resolving changes at the end

## Working with YOLO at current state

Install dependencies 

`mvn clean install -DskipTests`

Run program

`mvn exec:java`