红星
0%

Posted on Edited on

apple silicon install stable diffusion webui 后运行速度特别慢，参照 https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/7453

下载新的webui-user.sh文件，其实就是增加了三行

1
2
3
4
export COMMANDLINE_ARGS="--skip-torch-cuda-test --upcast-sampling --opt-sub-quad-attention --use-cpu interrogate"
venv_dir="venv-torch-nightly"
export TORCH_COMMAND="pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html"

然后重新执行webui.sh 后系统速度暴增。

Posted on Edited on

参考 Ivon的部落格

1
an asian girl on the beach
20240204001

改用新的提示词

1
1girl, asian, beach, ocean

然后将batch count改为4

20240204002

再算一次，但是结果变了？没错，即使提示词相近，AI绘图每次生图结果都是随机的。要维持上一次的结果并微调，你得保留每次算图的种子码(Seed，SD WebUI界面右下角，亦会写在文件名上)。

这时候注意一下右下角的内容

1
2
1girl, asian, beach, ocean
Steps: 20, Sampler: LMS, CFG scale: 7, Seed: 1655102876, Size: 512x560, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0
00013-1655102876

新测试,在seed 处填入：1655102876

1
2
3
4
1girl, asian, blue eyes, short hair, straw hat, standing, blue bikini, beach, ocean, orange sky

Steps: 20, Sampler: LMS, CFG scale: 7, Seed: 1655102876, Size: 512x560, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00014-1655102876

要改变风格，可以在提示词加入「风格」的提示词。这方面的提示词有：相片(photoshop)、3D建模(3d model)、装饰艺术(art deco)、石像(stone sculpture)等风格

1
2
3
4
5
a stone sculpture of 1girl, asian, blue eyes, beach, ocean
Steps: 20, Sampler: DPM++ 2M SDE, CFG scale: 7, Seed: 1822157101, Size: 512x512,
Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

Time taken: 14.5 sec.
00019-1822157101

风格是很难界定的东西呢，一般来说我们会图片像哪个艺术家的风格对吧？那么不如直接「召唤」他们吧！提示词可以包含某位画家的名字，AI会尝试模仿其风格。例如加上「慕夏的作品」，并加入「大师级作品」的风格提示词：

1
2
; 参考翻译：慕夏的作品，大师级作品，一个女孩，亚洲人，蓝眼睛，沙滩，海洋
artwork by Alfons Maria Mucha, masterpiece, 1girl, asian, blue eyes, beach, ocean

生图的结果就会变得像是慕夏的作品。顺带一提masterpiece这个提示词还蛮万用的，可以让作品维持一定水准。

00023-3397365016

当然还可以画特定动漫人物的「二创」，只要将名字和作品名称打上去就可以了。有名的动漫角色只要出现名字，不用特别指定身体特征就会生成原作风格了，例如指定《新世纪福音战士》的惣流·明日香·兰格雷：

1
2

asuka_langley_souryuu from neon_genesis_evangelion, masterpiece, 1girl, beach, ocean
00026-1538503611

再次提醒，在下提示词的时候，生成图片使用的模型也需纳入考量。譬如想要生成特定动漫人物，去Danbooru找到了英文姓名，加上提示词，但你却用画真人的Stable Diffusion去算，则AI可能根本就不会认得你说的角色是谁，反之亦然。Danbooru的标签风格跟一般图片的描述还是有差距的。

除非日后有搜罗全部网络图片的超大模型出现，否则下提示词须配合模型种类来决定。

如果AI怎样都生成不出你要的风格或人物，请考虑换个模型，或是 自行训练模型。

负向提示词 Negative Prompts

上面我们只写提示词，但下面的负向提示词的字段都是空白。但生图时常常还需要加入一些负向提示词，避免掉不好的结果。AI绘图有时不会一次就算出好结果，所以还需要加上负向提示词来控制，尤其是大批算图的时候更为重要。

负向提示词会加入一些常见的「不好」的图片特征，例如低画质、最糟品质、画家签名、模糊、浮水印

1
deformed, lowres, bad anatomy, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, artist name

不想看到的东西也可以加进去。例如不想看到裸露、兵器、血、猎奇的元素出现，就加入nsfw、weapon、blood、guro至负向提示词

1
nsfw, weapon, blood, guro, lowres, bad anatomy, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,signature, watermark, username, blurry, artist name

提示词并非越多越好，正向提示词以不超过150个词元(token)为原则；不过负向提示词可以尽量把不想看到的都塞进去。

Stable Diffusion WebUI独有语法

此节列出的语法为AUTOMATIC1111制作的SD WebUI独有，使用其他人制作的Stable Diffusion程序不见得适用。

关注度括号 Attention/emphasis

控制关注度的符号，增加算图时对该提示词的关注度(attention)。简单来说，括号就是你想强调的重点元素，括号越多，生成的结果越会符合括号里的提示词。

SD WebUI使用小括号控制关注度： ( ) 小括号层次越多权重越高(不加小括号为1倍，每加一层小括号乘以1.1倍)。例如，强调「蓝眼睛」，生成的结果就更高几率是蓝眼睛人物

1
2
3
4
asuka_langley_souryuu from neon_genesis_evangelion, masterpiece, 1girl, beach, ocean, ((blue eyes))
Steps: 20, Sampler: UniPC, CFG scale: 7, Seed: 1080937815, Size: 512x512, 
Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00027-1080937815

亦可以直接写明要增强几倍关注度，但其实一般情况下不用设太强，用一二层的小括号就够了。

1
2
3
4
asuka_langley_souryuu from neon_genesis_evangelion, masterpiece, 1girl, beach, ocean, ((blue eyes)),((yellow hair))
Steps: 20, Sampler: UniPC, CFG scale: 7, Seed: 2199430340, Size: 512x512,
Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00028-2199430340 
1
2
3
asuka_langley_souryuu from neon_genesis_evangelion, masterpiece, 1girl, beach, ocean, ((blue eyes)),((yellow hair)),(Green clothes)
Steps: 20, Sampler: UniPC, CFG scale: 7, 
Seed: 2630536656, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

这时候对颜色的控制已经错乱了，说明提示词混乱了。

00029-2630536656

提示词编辑 Prompt editing

指定在到哪一步数的时候切换提示词。此处的中括号跟权重无关。

语法为[提示词1:提示词2:要切换的步数]

例如，算图时设置20步，一开始算橘子，设置算到15步时切换成苹果

1
[orange : apple : 15] on table

这时候就是生成一个桔子

切换单字 Alternating Words

使用|代表在每个步数切换提示词，例如我要在算图时于「蓝眼睛」或「红眼睛」或「黑眼睛」之间切换

1
asian, woman, [blue eyes|red eyes|black eyes], beach, ocean

可组合性扩散 Composable Diffusion

此语法让AI依照权重生成二个不同的对象。

例如生成橘子与苹果，中间用大写的AND连接

1
orange AND apple

后面加上数字控制权重，初始值为1，数值低于0.1则无效。

1
orange :1.5 AND apple :2.5

此语法适合搭配 ControlNet和Latent Couple，可以准确控制图中对象的位置。

00116-2587609433

提示词范例

这里提供一些懒人包。负向提示词有些是通用的。

生成可爱的狗勾

让我们试着画几只柯基犬在草地上奔跑。

建议使用模型：Stable Diffusion、Realistic Vision

提示词

1
3 corgi dogs running on grass field

负向提示词

1
lowres, bad anatomy, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,signature, watermark, username, blurry, artist name

Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 2387293181, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00032-2387293181

生成高科技未来城市

建议使用模型：Stable Diffusion、Realistic Vision

提示词

1
city future, 8k, exploration, cinematic, realistic, unreal engine, hyper detailed, volumetric light, moody cinematic epic concept art, realistic matte painting, hyper photorealistic

负向提示词

1
lowres, bad anatomy, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,signature, watermark, username, blurry, artist name

Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 299709774, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00034-299709774

生成一台特斯拉车子

建议使用模型：Stable Diffusion、Realistic Vision

提示词

1
concept art,tesla car, aerodynamic, future

负向提示词

1
lowres, bad anatomy, text, error, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,signature, watermark, username, blurry, artist name

Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 3821788433, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00041-3821788433

生成动漫美少女

建议使用模型：Anything、Hentai Diffusion

随机画一个Fate/Grand Order的贞德

提示词

1
jeanne d'arc from fate grand order, 1girl, (best quality), (masterpiece), (high detail), ((full face)), sharp, ((looking at viewer)), ((detailed pupils)), (thick thighs), (((full body))), (large breasts)

负向提示词

1
lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts,signature, watermark, username, blurry, artist name

Steps: 20, Sampler: Euler, CFG scale: 7, Seed: 1480416086, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00052-1480416086

生成日韩风真人女孩

建议使用模型：ChilloutMix、Perfect World，并适时使用Japanese Doll的LoRA。

随机画一个日本女孩，不要色色

提示词

1
(a japanese woman), shiny skin, (ultra high res,photorealistic,realistic,best quality,photo-realistic), (((high detailed skin,visible pores))),(real person,photograph), (8k, raw photo, best quality, masterpiece),(1girl),photon mapping, radiosity, physically-based rendering,automatic white balance,(haunting smile,moist lips),watery eyes, (blush|cute and playful|adorable|thick bangs|beauty),((irises and pupils are rounded,the pupil reflects the surroundings,eyes are not the same size))

负向提示词

1
nude, lowres,blurry,simple background,jpeg artifacts,bad-artist,bad shadow,compressed image,low pixel,light spot, paintings,sketches,((monochrome)),((grayscale)),noise point,semi-realistic, 3d,render,cg,drawing,cartoon,anime,comic,username,watermark,signature,cropped,error,censored,text,stain, deformed iris,deformed pupils,deformed nail,deformed ear,deformed eye,deformed eyelid,collapsed eyeshadow, [excessive skin spots,excessive skin imperfections,skin blemishes,skin fold,rough skinstain skin],goosebumps,skin layering,axillary fold,facial contortion, (flawless face),trimming

Steps: 20, Sampler: DDIM, CFG scale: 7, Seed: 2228126458, Size: 512x512, Model hash: cc6cb27103, Model: v1-5-pruned-emaonly, Version: v1.7.0

00057-2228126458

Prompt:

(a japanese woman), shiny skin, (ultra high res,photorealistic,realistic,best quality,photo-realistic), (((high detailed skin,visible pores))),(real person,photograph), (8k, raw photo, best quality, masterpiece),(1girl),photon mapping, radiosity, physically-based rendering,automatic white balance,(haunting smile,moist lips),watery eyes, (blush|cute and playful|adorable|thick bangs|beauty),((irises and pupils are rounded,the pupil reflects the surroundings,eyes are not the same size))

20240207110110-1201812112

Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 1201812112, Size: 512x512, Model hash: 4199bcdd14, Model: revAnimated_v122EOL, Version: v1.7.0

5. 值得参考的提示词网站 #

若使用的是SD模型，想生成真人，请看 Voldy整理的历史上有名的艺术家。

若使用动漫风的模型请看 Danbooru图库的标签决定要下哪些提示词。

Prompt Generator和 NovelAI魔导书可以协助你组合提示词。

Civitai除了下载模型外，还有很多现成的提示词搭配模型的范例可以照抄。

Posted on Edited on

https://github.com/comfyanonymous/ComfyUI

Dependencies

Install the dependencies by opening your terminal inside the ComfyUI folder and:

1
2
conda activate comfyui
pip install -r requirements.txt

Run

1
python main.py --force-fp16

comfyUI examples

https://comfyanonymous.github.io/ComfyUI_examples/

这里包含所有comfyUI的例子，这里所有图片都包含可以被comfyUI加载的 metadata 信息

RealESRGAN_x4plus.pth

https://huggingface.co/lllyasviel/Annotators/blob/main/RealESRGAN_x4plus.pth

Posted on Edited on

https://developer.apple.com/metal/pytorch/

Anaconda

Setup

Apple silicon

1
2
curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh
sh Miniconda3-latest-MacOSX-arm64.sh

Install

Anaconda
1
conda install pytorch torchvision torchaudio -c pytorch-nightly
pip
1
pip3 install --pre torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/nightly/cpu

Verify

You can verify mps support using a simple Python script:

1
2
3
4
5
6
7
import torch
if torch.backends.mps.is_available():
    mps_device = torch.device("mps")
    x = torch.ones(1, device=mps_device)
    print (x)
else:
    print ("MPS device not found.")

The output should show:

1
tensor([1.], device='mps:0')

Posted on Edited on 
1
2
3
4
5
6
7
8
CREATE DATABASE LINK remote_db
CONNECT TO remote_username IDENTIFIED BY remote_password
USING '172.16.40.2:1521/HWDB';

select db_link,username,host from dba_db_links;



Posted on Edited on

3.x版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
yum install BackupPC

cd /usr/share/BackupPC/
chown backuppc:apache sbin/*
cd sbin

chmod u+s BackupPC_Admin

usermod -s /bin/bash backuppc

firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --reload

# /etc/BackupPC/config.pl
$Conf{CgiAdminUsers}     = 'backuppc';
$Conf{PingPath} = '/bin/ping';


htpasswd -c /etc/BackupPC/apache.users backuppc

zaq1XSW2

systemctl restart backuppc
 systemctl restart httpd

4.x

1
2
3
4
5
6
yum --enablerepo=extras -y install epel-release
yum --enablerepo=epel-testing -y install BackupPC-XS rsync-bpc

 yum groupinstall "Development Tools"


安装BackupPC-XS

1
2
3
4
5
6
7
8
 https://objects.githubusercontent.com/github-production-release-asset-2e65be/ 
tar zxf BackupPC-XS-0.62.tar.gz
cd BackupPC-XS-0.62

perl Makefile.PL
make
make test
make install

Install the following requisites for BackupPC version 4.3.1

1
2
3
4
5

yum -y install bzip2 httpd mod_perl par2cmdline perl-Archive-Zip perl-CGI \
perl-Compress-Raw-Zlib perl-Data-Dumper perl-Digest-MD5 perl-File-Listing \
perl-File-RsyncP perl-Net-FTP-AutoReconnect perl-Net-FTP-RetrHandle \
perl-Time-ParseDate perl-XML-RSS perl-version rrdtool samba-client

Lets set up some directories that we are going to use later.

My backuppc server has a raid10 array mounted at /data/. I am going to use this to store the data for backuppc. Lets create the backuppc folder on my raid. Note: this folder can grow to be extremely large.

1
mkdir -p /data/backuppc

Now lets create one more folder for BackupPC to store some web interface files.

1
mkdir -p /var/www/html/backuppc

And create the cgi-bin directory for backuppc

1
mkdir -p /var/www/cgi-bin/backuppc

Make note of those 2 directories. The installer will ask for them if we don’t explicitly set them below.

Create and setup the backuppc user.

Create group backuppc.

1
groupadd backuppc

Create backuppc user, and tell it where its home folder is. My backuppc user’s home folder is /opt/backuppc. Change this to whatever you’d like.

1
2
3
useradd --home-dir /opt/backuppc --create-home --shell /bin/bash --base-dir /opt/backuppc --gid backuppc backuppc


Set Permissions on the directories we created:

1
2
chown -R backuppc:backuppc /data/backuppc
chown -R backuppc:backuppc /opt/backuppc

Download the BackupPC tar.gz archive and extract it.

1
2
3
4
5
cd ~
wget https://github.com/backuppc/backuppc/releases/download/4.4.0/BackupPC-4.4.0.tar.gz
tar zxf BackupPC-4.4.0.tar.gz
cd BackupPC-4.4.0

Run the configure.pl script

Pay close attention to the paths in this command.

1
2
3
perl configure.pl --batch --cgi-dir /var/www/cgi-bin/backuppc \
--data-dir /data0/backuppc --hostname backuppc --html-dir /var/www/html/backuppc \
--html-dir-url /backuppc --install-dir /opt/backuppc

Alternatively, you can execute the perl script with no arguments and it will walk you through the install step by step. This is the preferred method if you are upgrading a version 3.X install. (Not supported by this guide)

1
perl configure.pl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
Ok, we're about to:

  - install the binaries, lib and docs in /usr/local/BackupPC
  - create the data directory /data/BackupPC
  - optionally install the cgi-bin interface
  - create/update the config.pl file /etc/BackupPC/config.pl
--> Do you want to continue? [y]?
Created /usr/local/BackupPC/bin
Created /usr/local/BackupPC/share
Created /usr/local/BackupPC/share/doc
Created /usr/local/BackupPC/share/doc/BackupPC
Created /usr/local/BackupPC/lib
Created /usr/local/BackupPC/lib/BackupPC
Created /usr/local/BackupPC/lib/BackupPC/CGI
Created /usr/local/BackupPC/lib/BackupPC/Config
Created /usr/local/BackupPC/lib/BackupPC/Lang
Created /usr/local/BackupPC/lib/BackupPC/Storage
Created /usr/local/BackupPC/lib/BackupPC/Xfer
Created /usr/local/BackupPC/lib/BackupPC/Zip
Created /usr/local/BackupPC/lib/Net
Created /usr/local/BackupPC/lib/Net/FTP
Created /data/BackupPC
Created /data/BackupPC/pool
Created /data/BackupPC/cpool
Created /data/BackupPC/pc
Created /etc/BackupPC
Created /var/log/BackupPC
Created /var/run/BackupPC
Installing binaries in /usr/local/BackupPC/bin
Installing library in /usr/local/BackupPC/lib
Making systemd and init.d scripts
Making Apache configuration file for suid-perl
Installing docs in /usr/local/BackupPC/share/doc/BackupPC
Installing cgi script BackupPC_Admin in /var/www/cgi-bin/backuppc
Installing config.pl and hosts in /etc/BackupPC

Ok, it looks like we are finished.  There are several more things you
will need to do:

  - Browse through the config file, /etc/BackupPC/config.pl,
    and make sure all the settings are correct.  In particular,
    you will need to set $Conf{CgiAdminUsers} so you have
    administration privileges in the CGI interface.

  - Edit the list of hosts to backup in /etc/BackupPC/hosts.

  - Read the documentation in /usr/local/BackupPC/doc/BackupPC.html.
    Please pay special attention to the security section.

  - Verify that the CGI script BackupPC_Admin runs correctly.  You might
    need to change the permissions or group ownership of BackupPC_Admin.
    If this is an upgrade and you are using mod_perl, you will need
    to restart Apache.  Otherwise it will have stale code.

  - BackupPC should be ready to start.  Don't forget to run it
    as user backuppc!  The installation also contains
    a systemd/backuppc.service script that can be installed so
    that BackupPC can auto-start on boot.  This will also enable
    administrative users to start the server from the CGI interface.
    See systemd/README.

Enjoy!

Apache config

I chose the CGI route (not S-CGI) and chose the directory /var/www/cgi-bin/backuppc for my CGI directory. I chose /var/www/html/backuppc as my images directory. You may need to set this path in the config if you ran the configure.pl script with no arguments.

Now we need to copy the Apache config into the apache conf.d directory.

1
cp httpd/BackupPC.conf /etc/httpd/conf.d/

Install the included systemd script:

1
2
3
4
cp systemd/backuppc.service /etc/systemd/system/
systemctl daemon-reload
systemctl start backuppc
systemctl enable backuppc

Create the backuppc authentication file.

1
2
3
htpasswd -c /etc/BackupPC/BackupPC.users backuppc

Note: this sets the username to backuppc (modify it to your liking)

Set permissions on this file.

1
chown backuppc:backuppc /etc/BackupPC/BackupPC.users

Edit the config.pl file in /etc/BackupPC/

1
vim /etc/BackupPC/config.pl

Check the image directory and image url variables.

1
2
$Conf{CgiImageDir} = '/var/www/html/backuppc';
$Conf{CgiImageDirURL} = '/BackupPc';

Also add backuppc as the administrative user: $Conf{CgiAdminUsers} = 'backuppc';

这时候不能往/etc/BackupPC目录写东西，

chcon -R -t httpd_sys_content_t /etc/BackupPC

查看备份内容

1
2
3
4
5
6
# 列表
BackupPC_ls -h bi-server -n 0 -s /data/bakdata /
BackupPC_ls -h erp-oracle -n 1 -s /backup/bakdata /

#查看文件
BackupPC_zcat -h erp-oracle -n 1 -s /backup/bakdata /navy_bak221220_full.log

Posted on Edited on

Linux查看CPU信息，机器型号，内存等信息

1. 系统

命令 含义
uname -a 查看内核/操作系统/CPU信息
head -n 1 /etc/issue 查看操作系统版本
cat /proc/cpuinfo 查看CPU信息
hostname 查看计算机名
lspci -tv 列出所有PCI设备
lsusb -tv 列出所有USB设备
lsmod 列出加载的内核模块
env 查看环境变量
lspci | grep -i nvidia 查看NVIDIA显卡

2. 资源

命令 含义
free -m 查看内存使用量和交换区使用量
df -h 查看各分区使用情况
du -sh <目录名> 查看指定目录的大小
grep MemTotal /proc/meminfo 查看内存总量
grep MemFree /proc/meminfo 查看空闲内存量
uptime 查看系统运行时间、用户数、负载
cat /proc/loadavg 查看系统负载

3. 磁盘和分区

命令 含义
mount column -t 查看挂接的分区状态
fdisk -l 查看所有分区
swapon -s 查看所有交换分区
hdparm -i /dev/hda 查看磁盘参数(仅适用于IDE设备)
dmesg grep IDE 查看启动时IDE设备检测状况

4. 网络

命令 含义
ifconfig 查看所有网络接口的属性
iptables -L 查看防火墙设置
route -n 查看路由表
netstat -lntp 查看所有监听端口
netstat -antp 查看所有已经建立的连接
netstat -s 查看网络统计信息

5. 进程

命令 含义
ps -ef 查看所有进程
top 实时显示进程状态

6. 用户

命令 含义
w 查看活动用户
id <用户名> 查看指定用户信息
last 查看用户登录日志
cut -d: -f1 /etc/passwd 查看系统所有用户
cut -d: -f1 /etc/group 查看系统所有组
crontab -l 查看当前用户的计划任务

7. 服务

命令 含义
chkconfig –list 列出所有系统服务
chkconfig –list | grep on 列出所有启动的系统服务

8. 程序

命令 含义
rpm -qa 查看所有安装的软件包

9. 查看CPU信息（型号）

  • CPU型号
    cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
  • 几颗核心
    cat /proc/cpuinfo | grep physical | uniq -c
  • 查看CPU模式
    getconf LONG_BIT
  • 查看CPU运算flags
    cat /proc/cpuinfo | grep flags | grep ' lm ' | wc -l
  • 完整看cpu详细信息
    dmidecode | grep 'Processor Information
  • 查看内存信息
    cat /proc/meminfo
  • 查看当前操作系统内核信息
    uname -a
  • 查看当前操作系统发行版信息
    cat /etc/issue | grep Linux
  • 查看机器型号
    dmidecode | grep "Product Name
  • 查看网卡信息
    dmesg | grep -i eth

10. GPU相关命令

  • 查看显卡信息
    lspci | grep -i vga
  • 若使用NVIDIA显卡
    lspci | grep -i nvidia
  • 查看显卡详情
    lspci -v -s 00:0f.0
  • 查看显存使用情况
    nvidia-smi
  • 周期性输出显卡使用情况
    watch -n 10 nvidia-smi
  • 查看cuda版本
    cat /usr/local/cuda/version.txt
  • 查看cudnn版本
    cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2

Posted on Edited on

开源oss

  • Minio：一个基于Apache License v2.0协议的开源对象存储服务器。Minio支持Amazon S3兼容API，可以用于构建云端的备份，CDN和存储库。

  • Ceph：一个分布式文件系统，可以在计算机集群上提供高性能的对象存储和文件系统服务。Ceph是一个开源项目，提供了一个名为RadosGW的对象存储服务器，支持Amazon S3和OpenStack Swift API。

  • GlusterFS：一个分布式文件系统，可以在计算机集群上提供高性能的文件存储服务。GlusterFS使用FUSE来将分布式文件系统挂载到本地文件系统上，可以提供类似于NFS或CIFS的文件共享服务。

Posted on Edited on

20231110-kettle学习

kettle用repositories

新建一个数据库的仓库，记着要把启动命令中增加 “-Dfile.encoding=UTF-8”,否则打开文件的时候会出现 Invalid byte 1 of 1-byte UTF-8 sequence.

然后新建的转换和作业就保存到数据库中

linux服务器操作

把pdi解压缩后复制到/usr/local/data-integration 目录

设置环境变量

1
export KETTLE_HOME=/usr/local/data-integration

把windows下做好的 repositories.xml 文件复制到/usr/local/data-integration/.kettle

目录下

查看repository:

1
2
3
4
5
./kitchen.sh -listrep
 List of repositories:
#1 : birepo [bi-databases]  id=KettleDatabaseRepository

查看所有的job

1
2
3
4
./kitchen.sh -rep:birepo  -user:admin -pass:admin   -listjobs
2023/11/10 17:40:59 - Kitchen - Start of run.
2023/11/10 17:40:59 - RepositoriesMeta - Reading repositories XML file: /usr/local/data-integration/.kettle/repositories.xml
zgcw

查看所有的trans

1
2
3
4
5
./pan.sh -rep:birepo  -user:admin -pass:admin   -listtrans
2023/11/10 17:43:09 - Pan - 开始运行.
2023/11/10 17:43:09 - RepositoriesMeta - Reading repositories XML file: /usr/local/data-integration/.kettle/repositories.xml
car_transfer_info
market

执行转换：

1
2
3
4
5
6
7
8
9
10
11
12
13
$ ./pan.sh -rep:birepo  -user:admin -pass:admin   -trans:market
2023/11/10 17:45:35 - Pan - 开始运行.
2023/11/10 17:45:35 - RepositoriesMeta - Reading repositories XML file: /usr/local/data-integration/.kettle/repositories.xml
2023/11/10 17:45:36 - market - 为了转换解除补丁开始  [market]
2023/11/10 17:45:36 - read_from_market.0 - Finished reading query, closing connection
2023/11/10 17:45:36 - read_from_market.0 - 完成处理 (I=443, O=0, R=0, W=443, U=0, E=0)
2023/11/10 17:45:37 - write_to_market.0 - 完成处理 (I=443, O=0, R=443, W=443, U=0, E=0)
2023/11/10 17:45:37 - Pan - 完成!
2023/11/10 17:45:37 - Pan - 开始=2023/11/10 17:45:36.601, 停止=2023/11/10 17:45:37.081
2023/11/10 17:45:37 - Pan - 0 秒后处理结束.
2023/11/10 17:45:37 - market -
2023/11/10 17:45:37 - market - 进程 read_from_market.0 成功结束, 处理了 443 行. ( - 行/秒)
2023/11/10 17:45:37 - market - 进程 write_to_market.0 成功结束, 处理了 443 行. ( - 行/秒)

执行转换

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ ./pan.sh -rep:birepo  -user:admin -pass:admin   -trans:car_transfer_info
2023/11/10 17:46:18 - Pan - 开始运行.
2023/11/10 17:46:18 - RepositoriesMeta - Reading repositories XML file: /usr/local/data-integration/.kettle/repositories.xml
2023/11/10 17:46:18 - car_transfer_info - 为了转换解除补丁开始  [car_transfer_info]
2023/11/10 17:46:19 - get_lastast_time_of_car_transfer.0 - Finished reading query, closing connection
2023/11/10 17:46:19 - get_lastast_time_of_car_transfer.0 - 完成处理 (I=1, O=0, R=0, W=1, U=0, E=0)
2023/11/10 17:46:19 - read_modifed_record.0 - Finished reading query, closing connection
2023/11/10 17:46:19 - read_modifed_record.0 - 完成处理 (I=17, O=0, R=1, W=17, U=0, E=0)
2023/11/10 17:46:19 - update_car_transfer_info.0 - 完成处理 (I=17, O=15, R=17, W=17, U=2, E=0)
2023/11/10 17:46:19 - Pan - 完成!
2023/11/10 17:46:19 - Pan - 开始=2023/11/10 17:46:18.994, 停止=2023/11/10 17:46:19.542
2023/11/10 17:46:19 - Pan - 0 秒后处理结束.
2023/11/10 17:46:19 - car_transfer_info -
2023/11/10 17:46:19 - car_transfer_info - 进程 get_lastast_time_of_car_transfer.0 成功结束, 处理了 1 行. ( - 行/秒)
2023/11/10 17:46:19 - car_transfer_info - 进程 read_modifed_record.0 成功结束, 处理了 17 行. ( - 行/秒)
2023/11/10 17:46:19 - car_transfer_info - 进程 update_car_transfer_info.0 成功结束, 处理了 17 行. ( - 行/秒)

执行job

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ ./kitchen.sh -rep:birepo  -user:admin -pass:admin   -job:zgcw
2023/11/10 17:54:09 - Kitchen - Start of run.
2023/11/10 17:54:09 - RepositoriesMeta - Reading repositories XML file: /usr/local/data-integration/.kettle/repositories.xml
2023/11/10 17:54:10 - zgcw - 开始执行任务
2023/11/10 17:54:11 - zgcw - 开始项[market]
2023/11/10 17:54:11 - market - Using run configuration [Pentaho local]
2023/11/10 17:54:11 - market - 为了转换解除补丁开始  [market]
2023/11/10 17:54:11 - read_from_market.0 - Finished reading query, closing connection
2023/11/10 17:54:11 - read_from_market.0 - 完成处理 (I=443, O=0, R=0, W=443, U=0, E=0)
2023/11/10 17:54:11 - write_to_market.0 - 完成处理 (I=443, O=0, R=443, W=443, U=0, E=0)
2023/11/10 17:54:11 - zgcw - 开始项[car_transfer_info]
2023/11/10 17:54:11 - car_transfer_info - Using run configuration [Pentaho local]
2023/11/10 17:54:11 - car_transfer_info - 为了转换解除补丁开始  [car_transfer_info]
2023/11/10 17:54:11 - get_lastast_time_of_car_transfer.0 - Finished reading query, closing connection
2023/11/10 17:54:11 - get_lastast_time_of_car_transfer.0 - 完成处理 (I=1, O=0, R=0, W=1, U=0, E=0)
2023/11/10 17:54:12 - read_modifed_record.0 - Finished reading query, closing connection
2023/11/10 17:54:12 - read_modifed_record.0 - 完成处理 (I=10, O=0, R=1, W=10, U=0, E=0)
2023/11/10 17:54:12 - update_car_transfer_info.0 - 完成处理 (I=10, O=9, R=10, W=10, U=1, E=0)
2023/11/10 17:54:12 - zgcw - 开始项[成功]
2023/11/10 17:54:12 - zgcw - 完成作业项[成功] (结果=[true])
2023/11/10 17:54:12 - zgcw - 完成作业项[car_transfer_info] (结果=[true])
2023/11/10 17:54:12 - zgcw - 完成作业项[market] (结果=[true])
2023/11/10 17:54:12 - zgcw - 任务执行完毕
2023/11/10 17:54:12 - Kitchen - Finished!
2023/11/10 17:54:12 - Kitchen - Start=2023/11/10 17:54:10.967, Stop=2023/11/10 17:54:12.271
2023/11/10 17:54:12 - Kitchen - Processing ended after 1 seconds.

执行job带参数

1
2
./kitchen.sh -rep:erp_kettle  -user:admin -pass:admin -parameter:start_date=2023-12-1 -job:guanbao

做第一个市场每日交易统计

新建表 transfer_daily_report

1
2
3
4
5
6
7
8
9
10
CREATE TABLE `transfer_daily_report` (
  `market_id` int(10) DEFAULT NULL COMMENT '市场编号',
  `day` DATE  DEFAULT NULL COMMENT '天',
  `vehicle_type` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '车辆类型',
  `amount` varchar(30) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '金额',
  `cnt` int DEFAULT NULL COMMENT '数量',
  `market` varchar(200) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '市场名称',
  `province` varchar(100) COLLATE utf8mb4_general_ci DEFAULT NULL COMMENT '省',
  KEY `p` (`market_id`,`day`,`vehicle_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci COMMENT='日统计（每个市场、每天、分类型统计）';

新建重构作业流程

第一步先清库,新建执行sql的任务

1
delete  from transfer_daily_report;

新建一个获取全部交易执行的语句

1
2
3
4
5
6
7
8
SELECT date(  handle_time) as day,
 SUM(vehicle_price) as amount , count(*) as cnt,
 i.market_id,m.name as market, m.province_name as province,
 i.vehicle_type
FROM zg_vehicle_transfer_info i inner join market m on m.id = i.market_id 
 where i.handle_time is not null 
GROUP BY i.market_id,day ,i.vehicle_type;

新增一个生成车辆分类的脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
//Script here
var vehicle_category= ""
if(vehicle_type.indexOf("客车")>=0){
vehicle_category="客车"
}
else if(vehicle_type.indexOf("轿车")>=0){
vehicle_category="客车"
}
else if(vehicle_type.indexOf("小型汽")>=0){
vehicle_category="客车"
}
else if(vehicle_type.indexOf("小型轿")>=0){
vehicle_category="客车"
}
else if(vehicle_type.indexOf("面包车")>=0){
vehicle_category="客车"
}
else if(vehicle_type.indexOf("小型越野")>=0){
vehicle_category="客车"
}

else if(vehicle_type.indexOf("货车")>=0){
vehicle_category="货车"
}
else if(vehicle_type.indexOf("载货")>=0){
vehicle_category="货车"
}
else if(vehicle_type.indexOf("重型")>=0){
vehicle_category="货车"
}
else if(vehicle_type.indexOf("货运")>=0){
vehicle_category="货车"
}
else if(vehicle_type.indexOf("摩托车")>=0){
vehicle_category="摩托车"
}
else if(vehicle_type.indexOf("二轮")>=0){
vehicle_category="摩托车"
}

最后添加一个表输出的节点。

新建日更新的作业

新增一个获取最近10天交易的sql

1
2
3
4
5
6
7
SELECT date(handle_time) as day,
 SUM(vehicle_price) as amount , count(*) as cnt,
 i.market_id,m.name as market, m.province_name as province,
 i.vehicle_type
FROM zg_vehicle_transfer_info i inner join market m on m.id = i.market_id 
 where  i.handle_time is not null and  i.handle_time > date_sub( now(),interval 10 day)
GROUP BY i.market_id,day ,i.vehicle_type;

添加生成车辆类别的脚本（同上）

最后增加一个插入或者更新表的节点

这里要根据市场、日期、车辆类型来更新。