2022-05-06 13:37:16 +08:00
|
|
|
# Cube Studio
|
|
|
|
|
|
2022-08-03 11:32:42 +08:00
|
|
|
English | [简体中文](README_CN.md)
|
|
|
|
|
|
|
|
|
|
### Infra
|
2022-06-17 10:47:27 +08:00
|
|
|
|
2022-08-03 16:58:23 +08:00
|
|
|
<img width="1442" alt="image" src="https://user-images.githubusercontent.com/20157705/182568155-f0d06046-8bfc-49dd-b283-720db0e556bc.png">
|
2022-06-17 10:47:27 +08:00
|
|
|
|
2022-08-03 11:38:35 +08:00
|
|
|
cube-studio is a one-stop cloud-native machine learning platform open sourced by Tencent Music, Currently mainly includes the following functions
|
2022-08-03 11:48:03 +08:00
|
|
|
- 1、data management: feature store, online and offline features; dataset management, structure data and media data, data label platform
|
|
|
|
|
- 2、develop: notebook(vscode/jupyter); docker image management; image build online
|
|
|
|
|
- 3、train: pipeline drag and drop online; open template market; distributed computing/training tasks, example tf/pytorch/mxnet/spark/ray/horovod/kaldi/volcano; batch priority scheduling; resource monitoring/alarm/balancing; cron scheduling
|
2022-10-10 14:26:22 +08:00
|
|
|
- 4、automl: nni, ray
|
2022-08-03 11:48:03 +08:00
|
|
|
- 5、inference: model manager; serverless traffic control; tf/pytorch/onnx/tensorrt model deploy, tfserving/torchserver/onnxruntime/triton inference; VGPU; load balancing、high availability、elastic scaling
|
|
|
|
|
- 6、infra: multi-user; multi-project; multi-cluster; edge cluster mode; blockchain sharing;
|
2021-10-21 16:16:19 +08:00
|
|
|
|
2022-08-03 11:32:42 +08:00
|
|
|
# Doc
|
2022-06-17 10:47:27 +08:00
|
|
|
|
|
|
|
|
https://github.com/tencentmusic/cube-studio/wiki
|
2022-06-13 23:24:49 +08:00
|
|
|
|
2022-08-03 11:32:42 +08:00
|
|
|
# WeChat group
|
2022-05-05 14:35:05 +08:00
|
|
|
|
2022-08-31 23:25:24 +08:00
|
|
|
learning、deploy、consult、contribution、cooperation, join group, wechart id luanpeng1234 remark`<open source>`, [construction guide](https://github.com/tencentmusic/cube-studio/blob/master/CONTRIBUTING.md)
|
2023-02-18 11:51:40 +08:00
|
|
|
|
2023-02-18 11:52:37 +08:00
|
|
|
<img border="0" width="20%" src="https://user-images.githubusercontent.com/20157705/219829986-66384e34-7ae9-4511-af67-771c9bbe91ce.jpg" />
|
2022-06-20 19:53:13 +08:00
|
|
|
|
2022-08-03 11:32:42 +08:00
|
|
|
# Job Template
|
2022-06-21 17:18:21 +08:00
|
|
|
|
2022-08-03 11:48:03 +08:00
|
|
|
tips:
|
|
|
|
|
- 1、You can develop your own template, Easy to develop and more suitable for your own scenarios
|
2022-06-21 17:18:21 +08:00
|
|
|
|
2022-08-03 11:32:42 +08:00
|
|
|
| template | type | describe |
|
2022-06-21 17:18:21 +08:00
|
|
|
| :----- | :---- | :---- |
|
2022-08-03 11:32:42 +08:00
|
|
|
| linux | base | Custom stand-alone operating environment, free to implement all custom stand-alone functions |
|
|
|
|
|
| datax | import export | Import and export of heterogeneous data sources |
|
2022-08-22 16:23:38 +08:00
|
|
|
| hadoop | data processing | hdfs,hbase,sqoop,spark client |
|
2022-08-03 11:32:42 +08:00
|
|
|
| sparkjob | data processing | spark serverless |
|
2022-08-22 16:23:38 +08:00
|
|
|
| volcanojob | data processing | volcano multi-machine distributed framework |
|
2022-08-03 11:32:42 +08:00
|
|
|
| ray | data processing | python ray multi-machine distributed framework |
|
|
|
|
|
| ray-sklearn | machine learning | sklearn based on ray framework supports multi-machine distributed parallel computing |
|
2022-08-22 16:23:38 +08:00
|
|
|
| xgb | machine learning | xgb model training and inference |
|
|
|
|
|
| tfjob | deep learning | Multi-machine distributed training of tensorflow |
|
|
|
|
|
| pytorchjob | deep learning | Multi-machine distributed training of pytorch |
|
|
|
|
|
| horovod | deep learning | Multi-machine distributed training of horovod |
|
|
|
|
|
| paddle | deep learning | Multi-machine distributed training of paddle |
|
|
|
|
|
| mxnet | deep learning | Multi-machine distributed training of mxnet |
|
|
|
|
|
| kaldi | deep learning | Multi-machine distributed training of kaldi |
|
2022-08-03 11:32:42 +08:00
|
|
|
| tfjob-train | model train | distributed training of tensorflow: plain and runner |
|
|
|
|
|
| tfjob-runner | model train | distributed training of tensorflow: runner method |
|
|
|
|
|
| tfjob-plain | model train | distributed training of tensorflow: plain method |
|
|
|
|
|
| tf-model-evaluation | model evaluate | distributed model evaluation of tensorflow2.3 |
|
|
|
|
|
| tf-offline-predict | model inference | distributed offline model inference of tensorflow2.3 |
|
2022-08-22 16:23:38 +08:00
|
|
|
| model-register | model service | register model to platform |
|
|
|
|
|
| model-offline-predict | model service | distributed offline model inference of framework |
|
|
|
|
|
| deploy-service | model service | deploy inference service |
|
|
|
|
|
| media-download | multimedia data processing | Distributed download of media files |
|
|
|
|
|
| video-audio | multimedia data processing | Distributed extraction of audio from video |
|
|
|
|
|
| video-img | multimedia data processing | Distributed extraction of pictures from video |
|
2023-04-07 13:52:57 +08:00
|
|
|
| object-detection-on-darknet | machine vision | object-detection with darknet yolov3 |
|
2021-10-25 19:17:06 +08:00
|
|
|
|
2022-08-03 11:32:42 +08:00
|
|
|
# Deploy
|
2022-06-02 15:45:49 +08:00
|
|
|
|
2022-08-03 11:32:42 +08:00
|
|
|
[wiki](https://github.com/tencentmusic/cube-studio/wiki/%E5%B9%B3%E5%8F%B0%E5%8D%95%E6%9C%BA%E9%83%A8%E7%BD%B2)
|
2022-06-02 15:45:49 +08:00
|
|
|
|
2022-06-21 17:18:21 +08:00
|
|
|
|
2022-06-02 15:45:49 +08:00
|
|
|
|
2022-06-07 09:45:51 +08:00
|
|
|
|
2022-08-19 15:33:26 +08:00
|
|
|
# Company
|
|
|
|
|
|
2023-03-07 17:57:48 +08:00
|
|
|
|
