Quartus 18.1 fitter crash

[EugeneF]

Sometimes (after minor changes) during project compilation  fitter crashes with an error:

Error:
Internal Error: Sub-system: VPR20KMAIN, File: /quartus/fitter/vpr20k/vpr_common/cluster_legality_pincount.c, Line: 1332
Internal Error
Stack Trace:
0x573ec0: vpr_qi_jump_to_exit + 0x70 (fitter_vpr20kmain)
0xf20b7: vpr_exit_at_line + 0x97 (fitter_vpr20kmain)
0xf2100: handle_assertion_failure_stripped + 0x30 (fitter_vpr20kmain)
0x930e9: cl_legality_pincount_block_list_legal_for_inputs_and_outputs + 0x239 (fitter_vpr20kmain)
0x69470: l_block_list_feasible_with_specified_inputs + 0x150 (fitter_vpr20kmain)
0x69ccb: l_block_list_feasible_with_default_inputs_vpr_engine + 0xcb (fitter_vpr20kmain)
0x6646f: cl_block_list_and_first_ale_position_feasible_with_default_inputs + 0x1cf (fitter_vpr20kmain)
0x4a33f: l_anneal_try_ale_swap + 0x23f (fitter_vpr20kmain)
0x4a00f: l_anneal_try_pair_swaps + 0x20f (fitter_vpr20kmain)
0x4820f: l_anneal_prl_try_pair_swaps + 0x9f (fitter_vpr20kmain)
0x19e825: l_smid_job_thread_fn + 0x75 (fitter_vpr20kmain)
0x5c49cd: l_thread_start_wrapper + 0x3d (fitter_vpr20kmain)
0x1467e: msg_thread_wrapper + 0x6e (CCL_MSG)
0x16660: mem_thread_wrapper + 0x70 (ccl_mem)
0x2791: thr_thread_begin + 0xa1 (ccl_thr)
0x1570c: BaseThreadInitThunk + 0xc (kernel32)
0x5385c: RtlUserThreadStart + 0x1c (ntdll)

End-trace

Executable: quartus
Comment:
None

System Information
Platform: windows64
OS name: Windows 7
OS version: 6.1

Quartus Prime Information
Address bits: 64
Version: 18.1.0
Build: 625
Edition: Standard Edition

Device: Cyclone V  5CEFA9F23I7

Deletting db doesn't help. Different fitter settings also doesn't work. 

[SyafieqS]

Hi Eugene,

I can see you are using EOL Window 7 which might be hard to support. Is it possible to migrate to Win 10 and see if this resolve? 

[SyafieqS]

May I know if previous reply is feasible?

[EugeneF]

Sorry for the late answer. I tried to check it on computer with Win 10 a also upgrade Quartus to 21.1. But the problem remains the same. Nothing changes.

Problem Details
Error:
Internal Error: Sub-system: VPR20KMAIN, File: /quartus/fitter/vpr20k/vpr_common/cluster_legality_pincount.c, Line: 1332

Internal Error

Stack Trace:

   0x5d30d0: vpr_qi_jump_to_exit + 0x80 (fitter_vpr20kmain)

   0x106cda: vpr_exit_at_line + 0x8a (fitter_vpr20kmain)

   0x106d20: handle_assertion_failure_stripped + 0x30 (fitter_vpr20kmain)

    0x9fa65: cl_legality_pincount_block_list_legal_for_inputs_and_outputs + 0x255 (fitter_vpr20kmain)

    0x73ed0: l_block_list_feasible_with_specified_inputs + 0x150 (fitter_vpr20kmain)

    0x7473a: l_block_list_feasible_with_default_inputs_vpr_engine + 0xca (fitter_vpr20kmain)

    0x7152e: cl_block_list_and_first_ale_position_feasible_with_default_inputs + 0x1ce (fitter_vpr20kmain)

    0x51da8: l_anneal_try_ale_swap + 0x208 (fitter_vpr20kmain)

    0x51ab1: l_anneal_try_pair_swaps + 0x211 (fitter_vpr20kmain)

    0x4fd6f: l_anneal_prl_try_pair_swaps + 0x9f (fitter_vpr20kmain)

   0x1bf7e5: simd_job_wait + 0x95 (fitter_vpr20kmain)

    0x50347: l_anneal_prl_perform_temperature + 0x597 (fitter_vpr20kmain)

    0x4c447: anneal_clustering + 0x657 (fitter_vpr20kmain)

    0x4585f: cl_build_cluster_of_block_from_scratch + 0x75f (fitter_vpr20kmain)

    0x39996: do_clustering + 0x4e6 (fitter_vpr20kmain)

    0x7e126: cl_flow_pack_to_cbes_2 + 0x816 (fitter_vpr20kmain)

    0x7d816: cl_flow_pack_to_cbes + 0x96 (fitter_vpr20kmain)

   0x297abc: l_do_clustering_phase + 0x18c (fitter_vpr20kmain)

   0x296e14: aa_flow_place + 0x64 (fitter_vpr20kmain)

   0x296aec: aa_flow_fit + 0x11c (fitter_vpr20kmain)

   0x5d27e9: VPR_QI_FACADE::vpr_qi_main + 0x69 (fitter_vpr20kmain)

    0x3ce20: fitapi_run_vpr + 0x90 (fitter_fitapi)

    0x207ff: FSV_EXPERT_BASE::run_vpr + 0x13f (fitter_fsv)

    0x20229: FSV_EXPERT_BASE::place_and_route + 0x139 (fitter_fsv)

    0x1f677: FSV_EXPERT_BASE::invoke_fitter + 0x617 (fitter_fsv)

    0x1d282: fsv_execute + 0x22 (fitter_fsv)

     0xf01c: fmain_start + 0x8ac (FITTER_FMAIN)

     0x36ab: qfit_execute_fit + 0x1c7 (comp_qfit_legacy_flow)

     0x48d8: QFIT_FRAMEWORK::execute + 0x2b4 (comp_qfit_legacy_flow)

     0x1afb: qfit_legacy_flow_run_legacy_fitter_flow + 0x1c3 (comp_qfit_legacy_flow)

    0x14640: TclInvokeStringCommand + 0xf0 (tcl86)

    0x16442: TclNRRunCallbacks + 0x62 (tcl86)

    0x17c4d: TclEvalEx + 0x9ed (tcl86)

    0xa6a8b: Tcl_FSEvalFileEx + 0x22b (tcl86)

    0xa5136: Tcl_EvalFile + 0x36 (tcl86)

    0x127c0: qexe_evaluate_tcl_script + 0x340 (comp_qexe)

    0x11a6d: qexe_do_tcl + 0x2fd (comp_qexe)

    0x16c9e: qexe_run_tcl_option + 0x5ee (comp_qexe)

    0x369f1: qcu_run_tcl_option + 0xdd1 (comp_qcu)

    0x16599: qexe_run + 0x309 (comp_qexe)

    0x17641: qexe_standard_main + 0xc1 (comp_qexe)

     0x2092: qfit2_main + 0x82 (quartus_fit)

    0x13638: msg_main_thread + 0x18 (CCL_MSG)

    0x1494e: msg_thread_wrapper + 0x6e (CCL_MSG)

    0x18520: mem_thread_wrapper + 0x70 (ccl_mem)

    0x12df1: msg_exe_main + 0xa1 (CCL_MSG)

     0x2c88: __scrt_common_main_seh + 0x11c (quartus_fit)

    0x17033: BaseThreadInitThunk + 0x13 (KERNEL32)

    0x526a0: RtlUserThreadStart + 0x20 (ntdll)

End-trace

Executable: quartus
Comment:
None

System Information
Platform: windows64
OS name: Windows 10
OS version: 10.0

Quartus Prime Information
Address bits: 64
Version: 21.1.1
Build: 850
Edition: Standard Edition

[SyafieqS]

Is it possible to attach the design in qar for me to reproduce?

This seem to me a legit bug. we need a design to further investigate and possibly provide a workaround.

[EugeneF]

Unfortunately I can not send a design because of our company policy. 

I try to change a design and compiler settings  to find exact cause of error, but I still don't succeed with that. 

Can you help me to find possible error causes, according to file and functions where it happens, from the report?

[SyafieqS]

If the design cannot be shared, is small test design possible to be created at your end? replicating the issue.

Or you can send your  compilation database without sending the source design files? may follow below link on how to do that.

https://www.intel.com/content/www/us/en/support/programmable/articles/000076181.html

Quartus database at least will give some hint to move this forward.

[EugeneF]

Small parts of the design doesn't reproduce the issue. Only all design gives that error.

I've attached database, maybe it helps. 

[SyafieqS]

Hello,

Thanks for the database. I am checking this with engineering team. Will get back to you once there is any update

[SyafieqS]

I am still pending reply regarding the issue. Might take sometimes due to holiday seasons.

[SyafieqS]

Hello Eugene,

I am pending from developer and it has been assigned to right person.

Will let you know if there is any update.

[SyafieqS]

Hello,

This is still pending from developer for update.

May I know how urgent this issue is?

Is this internal error total roadblock your development?

[EugeneF]

Hello. It is not a total block, because we are able to find some compilable variant by making some random project changes. But that significantly increase development time after any project upgrade. So we are still waiting for the solution.

[SyafieqS]

Hi Eugene,

Understood. Below are some updates and request answer from developer. Let me know if that is not possible

Update: This IE is triggered when clusterer is determining if the LAB is legal by counting the inputs. The code assumes the input counts should be back to zero when all blocks are removed, but that's not the case for this design.

Unfortunately, this design seems to hit a rare corner case. Without the actual design that contains the specific LAB that triggers the IE, it is hard to identify the root cause of this bad behavior. And without this information, there are no simple/safe workarounds to bypass this IE.

Questions:

Have this design ever compiled successfully before?

Does you try to compile the design using different seeds?

Does you see any User Error/Warning messages before the IE?

Are there any special netlist topology? (self feeding loops, special location constraints, etc)

Is it possible to to pass in a post-mapped database (archived qar file) without the source, so that we can run quartus_fit at our end?

[EugeneF]

Hello!

Yes, sometimes this design compilled succesfully. Then, after some minor changes, it compiles with this error. Then, after some other changes in other places (adding some extra registers for example) it again compiles succesfully, ets... It can be a "random" changes in different places, not connected to some particular place or block of the design.  

Different seeds doesn't affect to the error. But different optimization settings (performance/power/area) sometimes help to eliminate error. (Or sometimes get it back)

There are no any Error messages before IE. Only some Warnings about severall unconstrained internall clocks in project.

There a no special location constraints. Only location for input/output pins.

How to create that post-mapped database?

[SyafieqS]

How to create that post-mapped database?

• There is KDB for this but for some reason it is offline, thus I post it here. Let me know if this is not possible.

Method 1: Using a Quartus II Archive File

Use the following procedure to export a Quartus II design database that does not include design source files. This method creates one archive file that someone else can use to recreate the project and compilation results without the source HDL files.

1. In the Quartus II software, from the Project menu, choose Archive Project.

2. Enter an Archive file name. The default is the current revision name for the project.

3. Under Include the following optional database files, select one of the following options:

o If the option is available for your device family, select Version-compatible database files (For future versions of the Quartus II software). This option is typically not available for devices that include any preliminary information in the Quartus II software (that is, the newest device families).

o If the above option is not available, select Compilation and simulation database files (For current versions of the Quartus II software). Note that this archive must be opened in the same Quartus II software version if you want to preserve the compilation results from the database.

4. Click Add/Remove Files.

5. Highlight the design files that do you not want to include in the archive, and click Remove. To highlight multiple files at once, hold the Ctrl key and click on each one, or hold the Shift key to highlight a set of files. You can also sort by file type by clicking on the Type column header; this allows you to find all the HDL files easily. Note that you can also remove other large files from the archive to reduce the archive file size, such as .pof, .sof, .pin, and .rpt files. Do not remove any files from the db directory.

6. Click OK in the Archive Complete dialog box that appears. If you chose to create a version-compatible database, by default the archive process creates a copy of the database in a directory called export_db.

7. You can now send out the software-generated archive file with the name <archive file name>.qar.

[SyafieqS]

Eugene,

Did you manage to follow the step that I gave? Let me know if there is any update.

[EugeneF]

Hello!

Sorry for the late reply. 

I am sending a compilation database archive file. Created in Quartus 20.1.

Hope it helps. 

[SyafieqS]

Eugene,

Thank you. I have passed the qar for further investigation. Will let you know if there is any update.

[SyafieqS]

Hi Eugene,

I am pending reply from engineering regarding this. Will update you once there is any

[IntelSupport · Accepted solution]

Hi Eugene,

Please let me know if you are able to resolve this using below suggestion.

Update:  

I am able to pin down the IE is triggered during heuristic accounting of input pins to a cluster. This heuristic counting is used to generate packing that give more flexibility for router to resolve local congestions for outlier designs, so it should be generally safe to disable this. I have tried the INI to disable this algorithm and the design is able to compile successfully.

Here is the encrypted INI (copy and paste this ini in your qsf) for 21.1 std (encrypted INI support is only avaliable for 20.1std or newer) then recompile.

ini_password = 0f56f88f5713676e261178731ef27f42b49cd891b355d81b400032576444751330035221150003342320131030052576