Can't lunch jupyter lab

[kunger]

I'm trying to lunch jupyter lab but I'm getting some errors.

Event log

Server requested

Cluster job running... waiting to connect

Spawn failed: Server at "idc-beta-batch-pvc-node-09:36629/user/u40596489597437f04ca6f641375ab67/" didn't respond in 120 seconds

my userid is u40596489597437f04ca6f641375ab67

[Thivagar_Intel]

Hi kunger,

Thank you for reaching out to us.

Please relaunch Jupyter Lab through the Training and Workshops section on the Intel Developer Cloud Console page and inform us if the issue persists.

If the issue continues, kindly follow the steps outlined below:

1. Access the Intel Developer Cloud Console page using a different browser or in an incognito window.

2. Clear the caches and cookies of the browser.

If the issue persists despite the suggested troubleshooting steps, please provide the following details for further assistance:

1. A complete screenshot of the error, including the URL.

2. Specify the country from which you are connecting.

Regards,

Thivagar

[kunger]

I've tried changing browsers and clearing browser cache and cookies

But it still show an error

[Zulkifli_Intel]

Hi kunger,

We were informed that the Slurm cluster had maintenance planned for Monday 26th February from 9 AM to 5 PM PDT. Slurm cluster for Training and Workshop will be down and all jobs will be stopped before 9:00 AM Monday.

Now the downtime has been completed and Jupyter Notebook is accessible again. Could you please relaunch the JupyterLab? Let us know if the issue persists.

Regards,

Zulkifli

[kunger]

i just try relaunch JupyterLab, but the issue still exists.

Maybe you can reset the account and try to fix this issue?

[Thivagar_Intel]

Hi kunger,

We apologize for any inconvenience you are currently experiencing. To further troubleshoot the issue, could you please provide the URL to your Jupyter Hub and let us know the country which you are currently connecting from.

Regards,

Thivagar

[kunger]

Jupyter Hub: https://idcbetabatch.eglb.intel.com/hub/spawn-pending/u40596489597437f04ca6f641375ab67

i'm connect from japan

by the way, i have tried use vpn to connect

[mcwrinn0]

Having the same issue reported by kunger:

Spawn failed: Server at http://idc-beta-batch-pvc-node-14:51455/user/ua789d607256d4359383062949953bcb/ didn't respond in 120 seconds

Worked through the same process earlier recommended (change browsers, etc), with no success. Help appreciated.

[Thivagar_Intel]

Hi kunger,

Thank you for providing the requested information. We have informed the development team about the issue for further investigation and will update you as soon as we have more details. Your patience is greatly appreciated.

Regards,

Thivagar

[marcosjrcwb]

Hi @kunger 

I'm also having the same problem. It's okay that they were "reserving it for a hackathom" but since they posted the message it never came back.
I have already cleared cookies, cache, different browsers in normal and incognito mode and always the same error of not loading Jupyter, regardless of the type of Training and Workshops available.
Worse, I have a lot of things that were already developed in the validation phase.

Another thing I noticed is that Python is version 3.9, and I've had several problems with libraries having to abort experiments.
Even using a virtualenv is limited, as you are faced with not having access to SUDO and being able to update a library.

Returning to the topic, @Thivagar_Intel when will we be able to use the training environment again even with the limitations I described of outdated versions?

http://idc-beta-batch-pvc-node-04:41485/user/u1b12e5b53846b170e1ea467d2ed6665/

[Kunger11]

Although the problem of not being able to launch jupyter lab is still unresolved, the python version issues you are experiencing can be resolved with the conda environment, which can be created by specifying the desired python version, and conda resolves most of the dependency issues as well.

[Thivagar_Intel]

Hi kunger,

Thank you for your patience. Your account has been re-enabled. Please relaunch Jupyter Lab through the Training and Workshops section on the Intel Developer Cloud Console page and let us know if the issue persists.

Upon investigation, it was suspected that your account was involved in crypto mining. Please take a moment to review our terms to ensure compliance and avoid any future conflicts.

Regards,

Thivagar

[kunger]

@Thivagar_Intel  Thanks for your assistance, I can now access my account. Regarding the crypto mining issue you mentioned, I can guarantee that my account has never been used for related operations. My account was disabled after using the CPU for LLM inference. In order to avoid another violation, I would like to reconfirm that using Training and Workshops jupyter lab for LLM inference is compliant

By the way, jupyter currently seems unable to use wget/curl to access services other than GitHub and huggingface, such as https://pytorch-extension.intel.com/release-whl/stable/xpu/us/, which is the official release of intel pytorch wheel. 

If you have restrictions on jupyterlab's network, is there a way for me to upload large files, such as private fine-tuned models?

[Thivagar_Intel]

Hi kunger,

Instead of using the wget/curl command, please utilize the Upload Files feature to upload large files such as private fine-tuned models to your node. You can locate the Upload Files feature under File Browser. Please refer to the image below.

As a friendly reminder, the maximum storage allocated for each node in JupyterLab is 30GB. Therefore, please cache the models to /tmp instead of your home directory, as recent models are generally multi-gigabyte in size, and downloading just one or two models can easily fill up the quota.

Regards,

Thivagar

[Thivagar_Intel]

Hi kunger,

This thread will no longer be monitored since we have provided a solution. If you need any additional information from Intel, please submit a new question.

Regards,

Thivagar