Items with no label
3335 Discussions

D435 can't reach 60FPS with PyRealsense2

SHuan76
Beginner
6,732 Views

Sorry for my poor English.

I want to record only RGB video with PyRealsense2 and opencv-python, the config is below:

config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 60)

But when I replayed the video I've recorded, I found that the number of frames is smaller than 60*the_time_I_spent_recording_in_seconds, in other words, the FPS is smaller than 60 (just 30 to 33).

My USB port is 3.2 and I prefer recording video with Python rather than Realsense Viewer.

Thnak you in advance for your apply.

0 Kudos
24 Replies
Eliza_D_Intel
Employee
4,927 Views
Hello SHuan76, Thank you for your interest in the Intel RealSense D435 camera. Could you please record a video with only 30 FPS and let us know what will be the number of frames received? Thank you, Eliza
0 Kudos
SHuan76
Beginner
4,927 Views

Hello, ElizaD_Intel,

I recorded a video with 30 FPS.

It received 1800 frames in 60.38 seconds (29.81FPS), so I think 30FPS works well.

0 Kudos
Eliza_D_Intel
Employee
4,927 Views
Hello SHuan76, Great news! If you want a higher number of FPS, you can use Intel RealSense Viewer to record the bag file. Thank you, Eliza
0 Kudos
SHuan76
Beginner
4,927 Views

Thanks for your reply, but I don't want to use Realsense Viewer cause the size of bag file is too big.

I ​prefer using Python code to record 60FPS video with Pyrealsense2 and opencv.

0 Kudos
Eliza_D_Intel
Employee
4,927 Views
Hello SHuan76, You can check out this article, where it is explained how to increase the number of FPS recorded with Python and OpenCV. https://www.pyimagesearch.com/2015/12/21/increasing-webcam-fps-with-python-and-opencv/ Thank you, Eliza
0 Kudos
SHuan76
Beginner
4,927 Views

Hello, ElizaD_Intel,

I tried that approach a few days ago, the drawback of this approach is that you can increase the FPS but can't control the FPS to be stable.

According to the datasheet of D435 (https://www.mouser.com/pdfdocs/Intel_D400_Series_Datasheet.pdf), D435 should support 60FPS well on resolution 640*480.

擷取.PNG

Thank you,

SHuan.

0 Kudos
Sahira_Intel
Moderator
4,927 Views
Hi SHuan, I apologize for the delay in responding. Are you just modifying the Python opencv_viewer_example to stream only RBG at 60 fps? When you stream at 6 or 15 fps, are you still getting an incorrect number of frames? I'll reproduce this in the lab and get back to you with my findings. Best Regards, Sahira
0 Kudos
SHuan76
Beginner
4,927 Views

Hello, sahira,

I use Pyrealsense2 to get rgb frames and use opencv-python to write these frames into a video.

30 fps works well, but I fail to get rgb frames in 60 fps.

0 Kudos
Sahira_Intel
Moderator
4,927 Views
Hi SHuan, Can you please provide your code? I'd like to run your exact program on my system. What FPS are you getting when using the RealSense Viewer while doing regular streaming without recording and when recording a bag file? Regards, Sahira
0 Kudos
SHuan76
Beginner
4,927 Views

Here's my code.

I run it on Jupyter Notebook. After running the code, you can see the RGB stream, press 'R' will start recording, and the stream will freeze, and press 'Q' will store the video.

**********************

import pyrealsense2 as rs

import numpy as np

import cv2

import datetime as dt

import threading as th

import multiprocessing as mp

import time

import os

 

class RGBRecordingClass(th.Thread):

  def __init__(self, out, queue):

    super(RGBRecordingClass, self).__init__()

    self.stop_flag = th.Event()

    self.stop_flag.set()

    self.out = out

    self.queue = queue

   

  def run(self):

    while self.stop_flag.isSet():

      if not self.queue.empty():

        self.out.write(self.queue.get())

    print('RGB Thread Exitted')

   

  def stop(self):

    self.stop_flag.clear()

     

pipeline = rs.pipeline()

config = rs.config()

config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

pipeline.start(config)

 

directory_path = '0429'

if not os.path.exists(directory_path):

  os.mkdir(directory_path)

 

rgb_queue = mp.Queue()

 

 

frame_count=0

start_record = False

rgb_video_name = directory_path+'/rgb.avi'

fourcc = cv2.VideoWriter_fourcc(*'XVID')

out = cv2.VideoWriter(rgb_video_name, fourcc, 30, (640, 480))

 

rgb_thd = RGBRecordingClass(out, rgb_queue)

rgb_thd.start()

 

 

 

try:

  while True:

 

    # Wait for a coherent pair of frames: depth and color

    frames = pipeline.wait_for_frames()

    color_frame = frames.get_color_frame()

     

     

    if not color_frame:

      print('not ready')

     

    # Convert images to numpy arrays

    color_image = np.asanyarray(color_frame.get_data())

 

     

    if start_record:

      rgb_queue.put(color_image)

      frame_count+=1

       

    # Show images

    if not start_record:

      cv2.imshow('frame', color_image)

 

     

     

    if cv2.waitKey(10) & 0xFF == ord('r'):

      start_record = True

      print('start recording...')

      print(str(dt.datetime.now()))

       

     

    if cv2.waitKey(10) & 0xFF == ord('q'):

      break

 

finally:

 

  # Stop streaming

  print('end')

  print('file saved')

  print(frame_count)

  print(str(dt.datetime.now()))

  cv2.destroyAllWindows()

  out.release()

  pipeline.stop()

  rgb_thd.stop()

0 Kudos
SHuan76
Beginner
4,927 Views

When I use RealSense Viewer to do streaming, 30 and 60 FPS both work well on 640*480 resolution, whether it's recording a bag file or not.

But the bag file is really too big, so it's excluded from my consideration.

0 Kudos
SHuan76
Beginner
4,927 Views

Supplement for the code above:

frame_count is the number of frames recorded, and it will show the time starting recording and finishing recording.

I calculate FPS by frame_count/(end time-start time)

0 Kudos
JesusG_Intel
Moderator
4,927 Views
Hello SirNight, Try setting the RS2_OPTION_AUTO_EXPOSURE_PRIORITY option to 0 to maintain constant FPS. If RS2_OPTION_AUTO_EXPOSURE_PRIORITY is set to 1, then FPS will fluctuate. Let us know if this solves your problem. You can test with a live stream on RealSense Viewer. Regards, Jesus Intel Customer Suppor
0 Kudos
SHuan76
Beginner
4,927 Views

擷取2.PNG擷取1.PNGIt's strange that when I enable auto exposure, the fps can reach 60 fps.

When I turn off auto exposure, the fps will drop to 32 fps.

 

 

0 Kudos
SHuan76
Beginner
4,927 Views

Besides, may you teach me how to set auto exposure with Pyrealsense2 (Python code)?

0 Kudos
JesusG_Intel
Moderator
4,927 Views
You want to set AUTO_EXPOSURE_PRIORITY, not AUTO_EXPOSURE. In your current settings, you may want to check your Exposure value. If it's high, the FPS will drop. Here is sample code for setting sensor options in Python: import pyrealsense2 as rs pipeline = rs.pipeline() config = rs.config() profile = pipeline.start(config) # Start streaming sensor_dep = profile.get_device().first_depth_sensor() print "Trying to set auto_exposure_priority" exp = sensor_dep.get_option(rs.option.auto_exposure_priority) print "exposure = %d" % exp print "Setting AUTO_EXPOSURE_PRIORITY to new value" exp = sensor_dep.set_option( rs.option.auto_exposure_priority, 0) exp = sensor_dep.get_option(rs.option.auto_exposure_priority) print "New AUTO_EXPOSURE_PRIORITY = %d" % exp profile = pipeline.stop You can find the full list of options in rs_option.h file. The Python wrapper exposes all these options like this: RS2_OPTION_EXPOSURE -> rs.option.exposure RS2_OPTION_ENABLE_AUTO_EXPOSURE -> rs.option.enable_auto_exposure RS2_OPTION_AUTO_EXPOSURE_PRIORITY -> rs.option.auto_exposure_priority In RealSense Viewer, you will find the control for Auto Exposure Priority under the RGB Controls menu. Regards, Jesus Intel Customer Support
0 Kudos
SHuan76
Beginner
4,927 Views

Hi Jesus G.,

I've tried both opening and closing auto exposure,

but the fps still stuck at about 32.

Also, whether I set rs.option.auto_exposure_priority to 0 or 1, the fps still can't rise to 60.

I really want to record a video in 60fps.

0 Kudos
JesusG_Intel
Moderator
4,927 Views
Have you re-tested with RealSense Viewer? If you record with Python then replay in RS Viewer, what FPS do you get? You can see the FPS by clicking the "i" icon on the top of the stream window.
0 Kudos
SHuan76
Beginner
4,927 Views

The file format I record the RGB video is .avi, so it can't be replayed with RS viewer.

The way how I count average FPS is (The number of frames catched)/(Total recording time).

I pasted the code 10 days ago above, replying to sahira.rizvi.

0 Kudos
JesusG_Intel
Moderator
4,478 Views
Mr. Huang, How you are you replaying the video and calculating the number of frames and total recording time? I want to ensure I reproduce exactly what you are doing. Regards, Jesus Intel Customer Support
0 Kudos
Reply