.vscode/settings.json

-{
-    "java.compile.nullAnalysis.mode": "automatic"
-}
\ No newline at end of file

UpdateEdits.md

+### Accepted feedback:
+- "Enhancement in the visual interface" by Manuel Segimon
+    - "I suggest a potential enhancement in the visual interface. Instead of adding a new object below the current one, consider a design that deletes the current one and adds a new one. This way, the page doesn’t get bigger the more objects you look at, improving the user experience."
+        - Our front end now should be able to do just that
+
+- "Confusing object selection in browser" by Seyed Reza Sajjadinasab
+    - "It is not clear what are the different 0-14 objects. Selecting between them didn't result in any change in the output."
+        - We are adding class display for object numbers to the front end
+
+- "install.txt is misleading" by Seyed Reza Sajjadinasab
+    - "It is said that the instruction provided is for "A Unix/Linux-based operating system", however, the installation source is a java library from a location in the lab machines. Using it on another Linux system will change "~/.bashrc" and cause problems for further usage of the OS.
+    The received error:
+    -bash: /ad/eng/opt/java/add_jdk17.sh: No such file or directory"
+        - This should be addressed now in the install.txt
+
+- "Ask user what type of object to detect" by Seyed Reza Sajjadinasab
+    - "You can specify a few types of objects that are detectable. It is not clear now what type of object is detected running mvn exec:java -Dexec.mainClass="yolo.YOLOTest" -Dexec.classpathScope="test". For example, why books are not detected as objects?" 
+        - The class names next to object names should clear this up
+
+### Rejected feedback:
+- "Web Interface improvement" by Tejas Thakur Singh
+    - "Add tabs or a compact grid view in order to fix the organization issues that come with having multiple views open"
+        - Sort of addressed already by Manuel's feedback
+
+- "Java implementation of the GUI" by Seyed Reza Sajjadinasab
+    - "It was required to do the majority of the project with Java. I think it's better to provide a Java version for the GUI as well instead of java script GUI."
+        - We focused on functionality of the GUI for the purposes of our 3D localization, and javascript is better suited with the necessary libraries to perform this task
+
+- "Showing the original video/picture" by Seyed Reza Sajjadinasab
+    - "The original video can be shown in the browser to make it easier to identify the significance of the tool and the process."
+        - Our reconstructed dense object map will take this into account. Furthermore, we are going to show the original scenario(s) the VSLAM is performed on during our demo. The final output on the front end is color mapped, so the objects are reletively visible in a 3D display of the scene
+
+- "Window size not adjusted" by Seyed Reza Sajjadinasab
+    - "The size of the window is not adjusted while using a browser to view the result."
+        - This seemed to be a sececondary issue to our main goal for the project
+
+- "Command line user interface for image and video" by Seyed Reza Sajjadinasab
+    - "It will be good if the user can pass the filename to the tool and it does the detection on the specified file."
+        - We have scripts available to perform this task on the backend, and we provide 2 RGBD datasets where users can perform VSLAM on for our functional front end. We also provide video captured using a monocular camera source that we can perform VSLAM on, but this dataset is not displayed in the front end due to our algorithms incompatibility with depth-less data
+
+### Additional changes
+- Improved object tracking algorithm
+- Improved VSLAM by incorportaing VSLAM RGBD
+- Ability to perform monocularr VSLAM on footage collected from arbitrary data source
+- The biggest issue that was raised was the GUI, due to its lack of clarity. We have completely overhauled the frontend, and improved the backend to provide better accuracy. The following list details how we did this:
+    1. Migrated from the THREE.js pointcloud plotting library to Plotly.js
+    2. Improved styling of the frontend

pom.xml

@@ -8,12 +8,11 @@
     <artifactId>vslam-objects</artifactId>
     <version>1.0</version>
 
-    <properties>
+     <properties>
         <maven.compiler.source>17</maven.compiler.source>
         <maven.compiler.target>17</maven.compiler.target>
         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
         <exec.mainClass>top.BackendJava</exec.mainClass>
-        <!-- <exec.mainClass>yolo.YOLODetector</exec.mainClass> -->
     </properties>
 
     <build>
@@ -126,4 +125,4 @@
         </dependency>
 
     </dependencies>
-</project>
+</project>
\ No newline at end of file

run_vslam_mono.sh

 #!/bin/bash
 
 # Define the script that runs the VSLAM implementation
+<<<<<<< HEAD
+<<<<<<< HEAD
+MATLAB_SCRIPT="/src/main/java/vslam/vslam_implementation_rgbd.m"
+
+# Check if the correct number of arguments was provided
+if [ "$#" -ne 1 ]; then
+    echo "Usage: $0 {tum_rgbd_dataset|imperial_college_london}"
+    exit 1
+fi
+
+# Determine which dataset to use based on the argument provided
+if [ "$1" = "tum_rgbd_dataset" ]; then
+    DATASET_NAME="tum_rgbd_dataset"
+elif [ "$1" = "imperial_college_london" ]; then
+    DATASET_NAME="imperial_college_london"
+else
+    echo "Invalid dataset name. Choose either 'tum_rgbd_dataset' or 'imperial_college_london'"
+    exit 2
+fi
+=======
+MATLAB_SCRIPT="/src/main/java/vslam/vslam_implementation.m"
+
+# Define the dataset name
+DATASET_NAME="self_made_dataset"
+>>>>>>> divider
+=======
 MATLAB_SCRIPT="/src/main/java/vslam/vslam_implementation.m"
 
 # Define the dataset name
 DATASET_NAME="self_made_dataset"
+>>>>>>> 197296f30ca7b0c49c8308bf3194348f11ab6a30
 
 # Navigate to the MATLAB script directory (assuming MATLAB can be called from command line)
 cd src/main/java/vslam

src/main/java/database/MongoDBInteraction.java

@@ -34,7 +34,6 @@ public class MongoDBInteraction {
         this.mongoClient = MongoClients.create(settings);
         this.database = mongoClient.getDatabase("Objects");
         this.objectCollection = database.getCollection("objectSets");
-        System.out.println("MongoDB Connection Established");
     }
 
 
@@ -46,10 +45,8 @@ public class MongoDBInteraction {
         try {
             Document doc = objectCollection.find().sort(new Document("index", -1)).first();
             if (doc == null) {
-                System.out.println("No document found.");
                 return null;
             } else {
-                System.out.println("Document found: " + doc.toJson());
                 return convertDocumentToObjectSet(doc);
             }
         } catch (Exception e) {
@@ -60,13 +57,11 @@ public class MongoDBInteraction {
 
     private ObjectSet convertDocumentToObjectSet(Document doc) {
         if (doc == null) {
-            System.out.println("Document is null, no conversion possible.");
             return null;
         }
 
         List<Document> pointSetDocs = doc.getList("objectSets", Document.class);
         if (pointSetDocs == null || pointSetDocs.isEmpty()) {
-            System.out.println("No point sets found in document.");
             return new ObjectSet();
         }
 
@@ -76,7 +71,7 @@ public class MongoDBInteraction {
             if (pointSet != null) {
                 objectSet.objects.add(pointSet);
             } else {
-                System.out.println("Failed to convert point set document: " + pointSetDoc.toJson());
+                System.out.println("Failed to convert point set document");
             }
         }
         return objectSet;
@@ -84,14 +79,15 @@ public class MongoDBInteraction {
 
     private PointSet convertDocumentToPointSet(Document doc) {
         if (doc == null) {
-            System.out.println("PointSet document is null.");
             return null;
         }
 
         // Using Integer.parseInt to safely convert String to Integer if necessary
         int idx;
+        String pred;
         try {
             idx = doc.get("setId") instanceof Integer ? (Integer) doc.get("setId") : Integer.parseInt((String) doc.get("setId"));
+            pred = (String) doc.get("predName");
         } catch (NumberFormatException e) {
             System.err.println("Invalid format for setId, must be an integer: " + doc.get("setId"));
             return null;
@@ -99,11 +95,10 @@ public class MongoDBInteraction {
 
         List<Document> pointsDocs = doc.getList("points", Document.class);
         if (pointsDocs == null) {
-            System.out.println("No points found in point set document.");
-            return new PointSet(idx);
+            return new PointSet(idx, pred);
         }
 
-        PointSet pointSet = new PointSet(idx);
+        PointSet pointSet = new PointSet(idx, pred);
         for (Document pointDoc : pointsDocs) {
             Point point = new Point(
                     pointDoc.getDouble("x").floatValue(),
@@ -130,6 +125,7 @@ public class MongoDBInteraction {
                     .append("B", p.B));
         }
         return new Document("setId", setId)
+                .append("predName", pointSet.getPred())
                 .append("points", pointsList);
     }
 

src/main/java/object_detection/ObjectDetector.java

@@ -3,6 +3,7 @@ package object_detection;
 import com.opencsv.exceptions.CsvValidationException;
 import database.MongoDBInteraction;
 import object_detection.types.*;
+import org.bytedeco.ffmpeg.avutil.Cmp_Const_Pointer_Const_Pointer;
 
 import java.io.File;
 import java.io.FileNotFoundException;
@@ -16,19 +17,28 @@ public class ObjectDetector {
 
     /**
      * The starting function that creates an object set, compiles informatino, and returns it
-     * @throws FileNotFoundException
+     * @throws FileNotFoundException : if files have not been created yet
      */
     public static void startProcess(String dataset) throws IOException, CsvValidationException {
 
         // for now, we can just set paths to the directories that hold keyframes and featurepoint CSVs
-
-        String bbox_dir_pth = "src/main/java/vslam/tum_rgbd/BoundedInfo";
-        String pose_dir_path = "src/main/java/vslam/rumrgbd/CameraPoses";
+        String bbox_dir_pth;
+        String pose_dir_path;
+        CameraIntrinsics intrinsics;
+        List<Point> pointCloud;
 
         if(dataset.equals("1")){
+            bbox_dir_pth = "src/main/java/vslam/tum_rgbd/BoundedInfo";
+            pose_dir_path = "src/main/java/vslam/tum_rgbd/CameraPoses";
+            intrinsics = new CameraIntrinsics("src/main/java/vslam/tum_rgbd/CameraIntrinsics.csv");
+            pointCloud = Downsampler.get_voxels("src/main/java/vslam/tum_rgbd/pointcloud.csv", 0.05F);
+        }
+        else{
             bbox_dir_pth = "src/main/java/vslam/imperial_london/BoundedInfo";
             pose_dir_path = "src/main/java/vslam/imperial_london/CameraPoses";
-        } 
+            intrinsics = new CameraIntrinsics("src/main/java/vslam/imperial_london/CameraIntrinsics.csv");
+            pointCloud = Downsampler.get_voxels("src/main/java/vslam/imperial_london/pointcloud.csv", 0.05F);
+        }
 
         // get files
         File[] bbox_CSVs = getDirFiles(bbox_dir_pth);
@@ -41,8 +51,6 @@ public class ObjectDetector {
         /* #################################################
         In the section below, we create a new ObjectSet, and iterate over each Keyframe
          ################################################## */
-        CameraIntrinsics intrinsics = new CameraIntrinsics("src/main/java/vslam/CameraIntrinsics.csv");
-        List<Point> pointCloud = Downsampler.get_voxels("src/main/java/vslam/pointcloud.csv", 0.05F);
         ObjectSet os = new ObjectSet(intrinsics, pointCloud);
 
         // iterate through each frame, create the frame, then process it
@@ -51,18 +59,7 @@ public class ObjectDetector {
             Frame f = new Frame(bbox_CSVs[i].getPath(), cp);
             os.processFrame(f);
             System.out.println("Processed frame " + i);
-            if(i == 5){
-                break;
-            }
-        }
-
-        Point[] pp = new Point[pointCloud.size()];
-        int i = 0;
-        for(Point p : pointCloud){
-            pp[i] = p;
-            i++;
         }
-        os.objects.add(new PointSet(0, pp));
 
         // update MongoDB
         MongoDBInteraction mdbi = new MongoDBInteraction();
@@ -90,9 +87,4 @@ public class ObjectDetector {
 
         return f_arr;
     }
-
-    public static void main(String[] args) throws IOException, CsvValidationException {
-        startProcess();
-    }
-
 }

src/main/java/object_detection/types/BoundingBox2D.java

@@ -17,6 +17,28 @@ public class BoundingBox2D {
                 && p.getY() <= (this.y + this.h);
     }
 
+    /* ##########
+        Getters
+    ########## */
+    public String getPredClass() {
+        return predClass;
+    }
+
+    public int getX() {
+        return x;
+    }
+
+    public int getY() {
+        return y;
+    }
+
+    public int getW() {
+        return w;
+    }
+
+    public int getH() {
+        return h;
+    }
 
     // members
     int x;

src/main/java/object_detection/types/BoundingBox3D.java

-package object_detection.types;
-
-public class BoundingBox3D {
-
-
-    /**
-     * This class represents a single object in 3D space, and holds the points that fall within it, and its bounds
-     * @param ps : the first pointset used to
-     */
-    public BoundingBox3D(PointSet ps){
-        this.ps = ps;
-    }
-
-
-    /* ###################
-        Members
-    ##################### */
-    private PointSet ps;
-}

src/main/java/object_detection/types/CameraIntrinsics.java

@@ -38,6 +38,25 @@ public class CameraIntrinsics {
         this.K = new DMatrixRMaj(ktemp);
     }
 
+    /* ##########
+        Getters
+    ########## */
+    public float[] getFocalLength() {
+        return FocalLength;
+    }
+
+    public float[] getPrincipalPoint() {
+        return PrincipalPoint;
+    }
+
+    public float[] getImageSize() {
+        return ImageSize;
+    }
+
+    public DMatrixRMaj getK() {
+        return K;
+    }
+
     /* ###############
         Members
      ############### */
@@ -46,8 +65,4 @@ public class CameraIntrinsics {
     float[] ImageSize;
     DMatrixRMaj K;
 
-
-    public static void main(String[] args) throws FileNotFoundException {
-        CameraIntrinsics intrinsics = new CameraIntrinsics("/Users/roku/IdeaProjects/group8/src/main/java/vslam/CameraIntrinsics.csv");
-    }
 }

src/main/java/object_detection/types/CameraPose.java

@@ -39,6 +39,16 @@ public class CameraPose {
         });
     }
 
+    /* ##########
+        Getters
+    ########## */
+    public DMatrixRMaj getR() {
+        return R;
+    }
+
+    public DMatrixRMaj getTranslation() {
+        return translation;
+    }
 
     /* ##################
         Members

src/main/java/object_detection/types/Frame.java

@@ -31,6 +31,12 @@ public class Frame {
         ArrayList<BoundingBox2D> res = new ArrayList<>();
         while (scanner.hasNextLine()) {
             String[] line = scanner.nextLine().split(",");
+            double confidence = Double.parseDouble(line[1]);
+            // only keep boxes with confidence > 88%
+            if(confidence < 0.82){
+                continue;
+            }
+
             // create the box accordingly
             int x = Integer.parseInt(line[2]);
             int y = Integer.parseInt(line[3]);
@@ -73,9 +79,9 @@ public class Frame {
 
             // check if point is visible in current frame
             if(projPoint.get(0,2) > 0){
-                if(x_div <= intrinsics.ImageSize[1]
+                if(x_div < intrinsics.ImageSize[1]
                     && x_div >= 0
-                    && y_div <= intrinsics.ImageSize[0]
+                    && y_div < intrinsics.ImageSize[0]
                     && y_div >= 0){
                     res.put(i, new Point2D((float) x_div, (float) y_div));
                 }
@@ -86,6 +92,21 @@ public class Frame {
         return res;
     }
 
+    /* ##########
+        Getters
+    ########## */
+    public String getBboxPath() {
+        return bboxpath;
+    }
+
+    public List<BoundingBox2D> getBoxes() {
+        return boxes;
+    }
+
+    public CameraPose getCamera() {
+        return camera;
+    }
+
     /* #########################
         Members
     ########################### */
@@ -93,14 +114,4 @@ public class Frame {
     List<BoundingBox2D> boxes;
     CameraPose camera;
 
-
-    public static void main(String[] args) throws IOException, CsvValidationException {
-        System.out.println("Trying projection:");
-        CameraIntrinsics ci = new CameraIntrinsics("src/main/java/vslam/CameraIntrinsics.csv");
-        List<Point> pc = Downsampler.get_voxels("src/main/java/vslam/pointcloud.csv", 0.05F);
-
-        CameraPose cp = new CameraPose("src/main/java/vslam/CameraPoses/Pose_0002.csv");
-        Frame f = new Frame("src/main/java/vslam/BoundedInfo/KeyFrame_0002.csv", cp);
-
-    }
 }

src/main/java/object_detection/types/ObjectSet.java

@@ -32,8 +32,6 @@ public class ObjectSet {
         // 2) Create a bitmap that holds the index of the bounding box that contains the pixel at bitmap[i][j]
         int[][] bitmap = new int[(int) intrinsics.ImageSize[0]][(int) intrinsics.ImageSize[1]];
 
-        System.out.println(bitmap.length);
-        System.out.println(bitmap[0].length);
         for(int b = 0; b < f.boxes.size(); b++){
             BoundingBox2D bbox = f.boxes.get(b);
             // for each box, write to all points in bitmap that fall in box
@@ -47,7 +45,7 @@ public class ObjectSet {
         // 3) Find all points that fall within each box, add point to candidate objects
         List<PointSet> candidates = new ArrayList<>();
         for(int i = 0; i < f.boxes.size(); i++){
-            candidates.add(new PointSet(i));
+            candidates.add(new PointSet(i, f.boxes.get(i).predClass));
         }
 
         for(Map.Entry<Integer, Point2D> entry : projM.entrySet()){
@@ -60,8 +58,8 @@ public class ObjectSet {
 
         // 4) Now that we have a full list of candidate objects, we can do object resolution by combining candidates with overlapping points
         for(PointSet c : candidates){
-            if(c.pset.size() > 2){
-                this.reconcileCandidate(c, 0.8);
+            if(c.pset.size() > 1000 && c.pset.size() < this.pointCloud.size()*0.75){
+                this.reconcileCandidate(c, 0.5);
             }
         }
 
@@ -76,18 +74,14 @@ public class ObjectSet {
             // calculate overlapping points, if beyond some threshold, find intersection of the sets
             int count = 0;
             for (Point p : obj.pset) {
-                for (Point cp : c.pset) {
-                    if (Point.equals(p, cp, 0.0001F)) {
-                        count++;
-                        break;
-                    }
+                if(c.pset.contains(p)){
+                    count++;
                 }
             }
 
-            if (count > (c.pset.size() * threshold)) {
+            if (count >= (obj.pset.size() * threshold)) {
                 //combine via intersection and return, since we know that we just combined those objects
                 obj.pset.retainAll(c.pset);
-                System.out.println("reconciling");
                 return;
             }
         }

src/main/java/object_detection/types/Point.java

 package object_detection.types;
 
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+
 public class Point {
 
 
@@ -39,13 +41,16 @@ public class Point {
             return false;
         }
         Point p = (Point) o;
-        return Point.equals(this, p, (float) 6);
+        return Point.equals(this, p, 0.0001F);
     }
 
     @Override
     public int hashCode() {
-        float res = this.x + this.y + this.z;
-        return Float.hashCode(res);
+        return new HashCodeBuilder(17, 37)
+                .append(this.x)
+                .append(this.y)
+                .append(this.z)
+                .toHashCode();
     }
     @Override
     public String toString() {