I recently encountered a problem when using dpgen to perform the model deviation. When paramiko downloads a large file, it will report the following error paramiko.ssh_exception.SSHException: Server connection dropped The error occurs when downloading the tar file of model_devi (123 Gb), which is consistent with the situation described in this article https://zhuanlan.zhihu.com/p/102372919. At present, the process of dpgen is to download the trajectory to the local and then obtain the candidate structures through analysis. If it can be changed to analyze the trajectory on the remote server, get the candidate structure and then download it locally, it can avoid downloading large trajectory files, greatly speeds up the process, and avoid the above errors. I hope the developers can consider my suggestion, thank you very much.

In order to speed up dpgen jobs, it may be practicable to accelerate FP stage by just skipping slow jobs or waiting finished asynchronously.

While running dpgen jobs, we found the duration time of FP phrase is a large part in each iteration. This is because the DFT computing for some candidates are really hard and time consuming, and we need to wait all those long tail to be finished before going to the next iteration. We found that the proportion of those candidates is very small, which may less than 1%.

So we may add new params "ratio_unfinished" to optimize the execution: if most of fp jobs have finished, we can directly discard the remaining jobs and go to the next iteration. We think it is acceptable since the ratio is very small. Further more, we can also enable async check to wait these jobs finished and download results asynchronously, but this step is need to be done in dpgen. Now we modify dpdispatcher as the first step.

Following is our test number, which can significantly saving time:

When I updated dpgen from 10.0 to 10.6, I encountered an ssh error

dpgen: 10.0 to 10.6

ssh erro for dpgen10.0

I solved the problem by changing the look_for_keys in client.py for 10.0 version, however, It doesnt work for 10.6 version.

ssh erro for dpgen10.6

Traceback (most recent call last): File "/data/jccao/app/deepmd2_1_5/bin/dpgen", line 8, in sys.exit(main()) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpgen/main.py", line 185, in main args.func(args) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpgen/generator/run.py", line 3642, in gen_run run_iter (args.PARAM, args.MACHINE) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpgen/generator/run.py", line 3607, in run_iter run_train (ii, jdata, mdata) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpgen/generator/run.py", line 598, in run_train submission = make_submission( File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpgen/dispatcher/Dispatcher.py", line 359, in make_submission machine = Machine.load_from_dict(abs_mdata_machine) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpdispatcher/machine.py", line 129, in load_from_dict context = BaseContext.load_from_dict(machine_dict) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpdispatcher/base_context.py", line 35, in load_from_dict context = context_class.load_from_dict(context_dict) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpdispatcher/ssh_context.py", line 350, in load_from_dict ssh_context = cls( File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpdispatcher/ssh_context.py", line 323, in init self.ssh_session = SSHSession(**remote_profile) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpdispatcher/ssh_context.py", line 44, in init self._setup_ssh() File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpdispatcher/utils.py", line 162, in wrapper return func(*args, **kwargs) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/dpdispatcher/ssh_context.py", line 166, in _setup_ssh ts.auth_password(self.username, self.password) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/paramiko/transport.py", line 1564, in auth_password return self.auth_handler.wait_for_response(my_event) File "/data/jccao/app/deepmd2_1_5/lib/python3.10/site-packages/paramiko/auth_handler.py", line 245, in wait_for_response raise e paramiko.ssh_exception.AuthenticationException: Authentication failed.

Details

Hi , guys.

I got a problem: Firstly, I run a job submission, then I stop it ( use Ctrl+C ). Secondly, I restart this same job, but the dpdispatcher cannot restart this jobs submission from last folder. It run a new calculation from a new folder.

Why cause this and how to fix it?

Thanks for any help.

Some environment info:

dpdispatcher: updated with branch master python: 3.8 "pbs": torque-6.1

Description

dpdispatcher works well on Machine with batch_type = slurm, shell but not PBS. I think maybe qstat -x in Line 53 has different behavior on my nodes and yours. https://github.com/deepmodeling/dpdispatcher/blob/60f5c90ef3b57dbbb270aeea8565bde74a53fdbd/dpdispatcher/pbs.py#L48-L78

1. If one job has been finished moments ago, and qstat return with nothing but return to stderr. Code in check_status() will go into Line 56, and return JobStatus.finished or JobStatus.terminated. And truth is I can get correct result (files downloaded, python exit with 0) after a while.
2. But the script will raise a RuntimeError

RuntimeError: job_state for job ...<json info of the job, not important>... is unknown

Locate the problem

qstat -x seems to return a somewhat XML format (maybe) like: (on shell, not from script)

<?xml version="1.0"?>
<Data></Data>

on my nodes.

I'm not sure whether I did anything wrong, or "pbs" not works the same way as yours. Perhaps @felix5572 knows this well. Maybe you are using PBS PRO but I'm using Torque.

By the way, I can also got job_state from my qstat -x return with label <job_state> but not simple .split()[-2]

<?xml version="1.0"?>
<Data>
<Job>
....
<job_state>C</job_state>
....
</Job>
</Data>

Traceback (most recent call last): File "submitted.py", line 6, in task0 = Task.load_from_json('json/task.json') AttributeError: type object 'Task' has no attribute 'load_from_json'

----here is the env I used

Name Version Build Channel

bcrypt 3.2.0 pypi_0 pypi ca-certificates 2021.5.30 h4653dfc_0 conda-forge certifi 2021.5.30 pypi_0 pypi cffi 1.14.6 pypi_0 pypi charset-normalizer 2.0.4 pypi_0 pypi cryptography 3.4.7 pypi_0 pypi dargs 0.2.6 pypi_0 pypi dpdispatcher 0.3.39 pypi_0 pypi idna 3.2 pypi_0 pypi libcxx 12.0.1 h168391b_0 conda-forge libffi 3.3 h9f76cd9_2 conda-forge ncurses 6.2 h9aa5885_4 conda-forge openssl 1.1.1k h3422bc3_1 conda-forge paramiko 2.7.2 pypi_0 pypi pip 21.2.4 pyhd8ed1ab_0 conda-forge pycparser 2.20 pypi_0 pypi pynacl 1.4.0 pypi_0 pypi python 3.8.10 hf9733c0_1_cpython conda-forge python_abi 3.8 2_cp38 conda-forge readline 8.1 hedafd6a_0 conda-forge requests 2.26.0 pypi_0 pypi setuptools 57.4.0 py38h10201cd_0 conda-forge six 1.16.0 pypi_0 pypi sqlite 3.36.0 h72a2b83_0 conda-forge tk 8.6.10 hf7e6567_1 conda-forge urllib3 1.26.6 pypi_0 pypi wheel 0.37.0 pyhd8ed1ab_1 conda-forge xz 5.2.5 h642e427_1 conda-forge zlib 1.2.11 h31e879b_1009 conda-forge

It is sometimes in need of executing command lines before or after task submitted to startup or shutdown necessary environment. While most issue with prepend script required like this might be solved by source a static script on remote server (which might also be difficult when remote server is not so stable.), scripts to be executed after tasks finished might be difficult under current dpdispatcher resource. Here in this PR, prepend_script and append_script parameters have been add to job.resource in form of List, in which a single line could be an item, like this:

"prepend_script": [
    "conda activate test_env",
    "export PATH=/path/to/package:$PATH",
    "send_an_email_to [email protected]"
    "sleep 1919810"
]

and the expected output:

conda activate test_env
export PATH=/path/to/package:$PATH
send_an_email_to [email protected]
sleep 1919810

Another little change is just add delay=True for logger to prevent the generation of dpdispatcher.log even if no log information output.

I found that re-running dpdispatcher would re-upload forward_files to the remote and replace it. But this can cause some problems:

• If forward_files is large, it will take a lot of time to retransfer files.
• Sometimes forward_files are rewritten during calculation, then the substitution will cause an error.

So it seems to me that since dpdispatcher checks for old submission, there is no need to re-upload forward_files.

After deleting a job group manually on bohrium, a NoneType error will be raised when submitting the job in the same path. To solve this problem, I add a check to NoneType and a note to help the user submit the job again. (Only the last commit in this pr is applied.)

For a large number of small files, compression can be very CPU-intensive and time-consuming. See https://github.com/deepmodeling/dpgen/issues/766. Add tar_compress in dict remote_profile. The archive will be compressed in upload and download if it is True. If not, compression will be skipped.

For some special queue or host, job submission might require waiting time after each single job submitted, to prevent crash when tasks were submitted together. Therefore, parameter wait_time is added to Resources, to support the special issue.

The default value of wait_time is 0, and it accepts a value of int. If set to a value larger than 0, it will sleep for wait_time seconds after each job submission.

Also, a condition is added for mirror_gitee action to prevent error message after pushing to the forked repository instead of the main one.

We have two batch job system (Torque and LSF). This example is run by RSA type key ( id_rsa ) for logging. But results are as below: In Torque, RuntimeError: Authentication failed, try to provide password In LSF, it works.

So I try connecting by paramiko.

import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect("Torque ip", username="user", key_filename="/home/user/.ssh/id_rsa")
ssh.connect("LSF ip", username="user", key_filename="/home/user/.ssh/id_rsa")

In Torque, ValueError: q must be exactly 160, 224, or 256 bits long paramiko In LSF, it works.

Summary

When I use dpgen run param.json machine.json to run a job, I frequently get errors: paramiko.ssh_exception.AuthenticationException: Authentication failed.

DeePMD-kit Version

2.1.1

TensorFlow Version

tf=2.5.0

Python Version, CUDA Version, GCC Version, LAMMPS Version, etc

python=3.8.5 cudatoolkit-11.3.1 gcc=7.5

Details

hello , dear Developers

When I use dpgen run param.json machine.json to run a job, I frequently find errors as follows: Please cite: Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E, DP-GEN: A concurrent learning platform for the generation of reliable deep learning based potential energy models, Computer Physics Communications, 2020, 107206.

Description

Traceback (most recent call last): File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/bin/dpgen", line 8, in sys.exit(main()) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpgen/main.py", line 185, in main args.func(args) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpgen/generator/run.py", line 3642, in gen_run run_iter (args.PARAM, args.MACHINE) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpgen/generator/run.py", line 3628, in run_iter run_fp (ii, jdata, mdata) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpgen/generator/run.py", line 3018, in run_fp run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _vasp_check_fin, File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpgen/generator/run.py", line 2985, in run_fp_inner submission = make_submission( File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpgen/dispatcher/Dispatcher.py", line 359, in make_submission machine = Machine.load_from_dict(abs_mdata_machine) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpdispatcher/machine.py", line 134, in load_from_dict context = BaseContext.load_from_dict(machine_dict) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpdispatcher/base_context.py", line 41, in load_from_dict context = context_class.load_from_dict(context_dict) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpdispatcher/ssh_context.py", line 350, in load_from_dict ssh_context = cls( File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpdispatcher/ssh_context.py", line 323, in init self.ssh_session = SSHSession(**remote_profile) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpdispatcher/ssh_context.py", line 44, in init self._setup_ssh() File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpdispatcher/utils.py", line 162, in wrapper return func(*args, **kwargs) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/dpdispatcher/ssh_context.py", line 166, in _setup_ssh ts.auth_password(self.username, self.password) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/paramiko/transport.py", line 1564, in auth_password return self.auth_handler.wait_for_response(my_event) File "/HOME/zhoujy/.conda/envs/deepmd-kit2.1.1/lib/python3.9/site-packages/paramiko/auth_handler.py", line 245, in wait_for_response raise e paramiko.ssh_exception.AuthenticationException: Authentication failed.

here is my machine.json，hostname、username and password is correct , about the fp , I want to run it on remote cluster: { "api_version": "1.0", "train": [ { "machine": { "context_type": "local", "batch_type": "Slurm", "machine_type": "Slurm", "local_root": "./", "_remote_root": "/data/run01/scz5616/HHM/dpmd-project/WFpro/S2tet/work", "remote_root": "/HOME/zhoujy/run/dp-test/work" }, "resources": { "module_list": [], "_source_list": [ "/data/run01/scz5616/HHM/dpmd-project/WFpro/S2tet/train.sh" ], "source_list": ["/HOME/zhoujy/run/dp-test/train.sh"], "cpu_per_node": 6, "number_node": 1, "gpu_per_node": 1, "queue_name": "gpu_c128", "_exclude_list": [], "_time_limit": "24:0:0", "group_size": 1 }, "command": "dp" } ], "model_devi": [ { "machine": { "context_type": "local", "batch_type": "Slurm", "machine_type": "Slurm", "local_root": "./", "_remote_root": "/data/run01/scz5616/HHM/dpmd-project/WFpro/S2tet/work", "remote_root": "/HOME/zhoujy/run/dp-test/work" }, "resources": { "_module_list": [], "_source_list": [ "/data/run01/scz5616/HHM/dp-test/lammps.sh" ], "cpu_per_node": 6, "number_node": 1, "gpu_per_node": 1, "queue_name": "gpu", "_exclude_list": [], "_time_limit": "23:0:0", "group_size": 1 }, "command": "lmp" } ], "fp": [ { "machine": { "context_type": "ssh", "batch_type": "Slurm", "_machine_type": "Slurm", "local_root": "./", "remote_root": "/public1/ws133/sc94566/zhou/work", "remote_profile": { "hostname": "36.103.203.6", "username": "sc94566@BSCC-A3", "port": 22, "password": "xxxxxxxxxxxxxxxxxxxx" } }, "resources": { "number_node": 1, "cpu_per_node": 64, "_custom_flags": [ "-p G1Part_sce" ], "queue_name": "amd_256", "_with_mpi": false, "source_list": [ "/public1/ws133/sc94566/zhou/env.sh" ], "_time_limit": "120:0:0", "_comment": "that's all", "group_size": 100 }, "command": "ulimit -s unlimited; srun -n 64 vasp_std" } ] }

After I updated the dpdispatcher version to 0.4.18, I got the following error when DPGEN performed the make_model_devi, which can be solved when downgrading dpdispatcher to 0.4.17.

2022-09-20 04:00:40,341 - INFO : job: 31fcd1c1d95b2fedff35615bf29adbc61e3057e5 315398 finished
INFO:dpgen:-------------------------iter.000007 task 02--------------------------
INFO:dpgen:-------------------------iter.000007 task 03--------------------------
INFO:dpgen:-------------------------iter.000007 task 04--------------------------
Traceback (most recent call last):
  File "/home/kwwan/.local/bin/dpgen", line 8, in <module>
    sys.exit(main())
  File "/home/kwwan/.local/lib/python3.8/site-packages/dpgen/main.py", line 185, in main
    args.func(args)
  File "/home/kwwan/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 3914, in gen_run
    run_iter (args.PARAM, args.MACHINE)
  File "/home/kwwan/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 3787, in run_iter
    run_model_devi (ii, jdata, mdata)
  File "/home/kwwan/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 1614, in run_model_devi
    run_md_model_devi(iter_index,jdata,mdata)
  File "/home/kwwan/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 1608, in run_md_model_devi
    submission.run_submission()
  File "/home/kwwan/software/Anaconda3/lib/python3.8/site-packages/dpdispatcher/submission.py", line 176, in run_submission
    self.generate_jobs()
  File "/home/kwwan/software/Anaconda3/lib/python3.8/site-packages/dpdispatcher/submission.py", line 340, in generate_jobs
    self.bind_machine(self.machine)
  File "/home/kwwan/software/Anaconda3/lib/python3.8/site-packages/dpdispatcher/submission.py", line 163, in bind_machine
    self.machine.context.bind_submission(self)
  File "/home/kwwan/software/Anaconda3/lib/python3.8/site-packages/dpdispatcher/ssh_context.py", line 389, in bind_submission
    self.block_checkcall(f"mv {old_remote_root} {self.remote_root}")
  File "/home/kwwan/software/Anaconda3/lib/python3.8/site-packages/dpdispatcher/ssh_context.py", line 537, in block_checkcall
    raise RuntimeError("Get error code %d in calling %s through ssh with job: %s . message: %s" %
RuntimeError: Get error code 1 in calling mv /data/home/scv3616/run/wankw/temp/dpmd_remote/447fbf8e9ee0ecc33a67e8f01f1847a2d3888f29 /data/home/scv3616/run/wankw/temp/dpmd_remote/5b3271c64c830aca6cfc836322191dc2482054ad through ssh with job: 5b3271c64c830aca6cfc836322191dc2482054ad . message:

I want dpdispatcher to handle those failed tasks via parameter ratio_unfinished, but I found that it doesn't work as expected when group_size exceeds 1. In this case, even if only one task in the group fails, all tasks in the group are deleted, which is unexpected.

I have added two examples below:

• https://docs.deepmodeling.com/projects/dpdispatcher/en/latest/examples/expanse.html
• https://docs.deepmodeling.com/projects/dpdispatcher/en/latest/examples/shell.html

Here, I would like to solicit more examples to add to the documentation. They should cover more job scheduling packages including PBS, LSF, Lebesgue, etc.